2020-10-09T17:01:22.597 INFO:root:teuthology version: 1.0.0-2cf4b81 2020-10-09T17:01:22.599 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --name aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi --owner scheduled_aemerson@teuthology --block --lock --archive /home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504 -- /tmp/teuthology-worker.3m3_06vh.tmp 2020-10-09T17:01:22.634 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-09T17:01:22.705 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504 branch: wip-investigate-datalog description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: aemerson@redhat.com first_in_suite: false job_id: '5510504' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-investigate-datalog ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4b457b3813ec2da916f501ce132355a10e79330a wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4b457b3813ec2da916f501ce132355a10e79330a rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-investigate-datalog sha1: 4b457b3813ec2da916f501ce132355a10e79330a owner: scheduled_aemerson@teuthology priority: 85 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 4b457b3813ec2da916f501ce132355a10e79330a sleep_before_teardown: 0 suite: rgw suite_branch: wip-investigate-datalog suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4b457b3813ec2da916f501ce132355a10e79330a tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2020-10-09_17:00:22 tube: smithi user: aemerson verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.24980 2020-10-09T17:01:22.705 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa; will attempt to use it 2020-10-09T17:01:22.707 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks 2020-10-09T17:01:22.707 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-10-09T17:01:22.709 INFO:teuthology.task.internal:Checking packages... 2020-10-09T17:01:22.732 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '4b457b3813ec2da916f501ce132355a10e79330a' 2020-10-09T17:01:22.732 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-10-09T17:01:22.733 INFO:teuthology.packaging:ref: None 2020-10-09T17:01:22.733 INFO:teuthology.packaging:tag: None 2020-10-09T17:01:22.733 INFO:teuthology.packaging:branch: wip-investigate-datalog 2020-10-09T17:01:22.734 INFO:teuthology.packaging:sha1: 4b457b3813ec2da916f501ce132355a10e79330a 2020-10-09T17:01:22.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-investigate-datalog 2020-10-09T17:01:22.915 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:01:22.916 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-10-09T17:01:22.924 INFO:teuthology.task.internal:no buildpackages task found 2020-10-09T17:01:22.924 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-10-09T17:01:22.970 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-10-09T17:01:22.971 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-09T17:01:23.019 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:01:33.037 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:01:43.047 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:01:53.061 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:03.077 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:13.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:23.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:33.118 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:43.132 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:02:53.147 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-10-09T17:03:03.165 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:03:13.182 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:03:23.199 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:03:33.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:03:43.235 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:03:53.250 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-10-09T17:04:03.266 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_aemerson@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504'} 2020-10-09T17:04:03.287 DEBUG:teuthology.lock.ops:locked ubuntu@smithi060.front.sepia.ceph.com, ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:04:03.341 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1375d75c80>, signals=[15]) 2020-10-09T17:04:03.527 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of ubuntu 18.04 2020-10-09T17:04:03.540 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of ubuntu 18.04 2020-10-09T17:04:03.816 INFO:teuthology.orchestra.console:Power off smithi060 2020-10-09T17:04:03.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-10-09T17:04:03.844 INFO:teuthology.orchestra.console:Power off smithi184 2020-10-09T17:04:03.845 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-10-09T17:04:03.855 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:03.867 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:07.881 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:07.892 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:11.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:12.008 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:12.206 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2020-10-09T17:04:12.218 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2020-10-09T17:04:12.308 INFO:teuthology.orchestra.console:Power on smithi060 2020-10-09T17:04:12.309 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-10-09T17:04:12.321 INFO:teuthology.orchestra.console:Power on smithi184 2020-10-09T17:04:12.321 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-10-09T17:04:12.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:12.345 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:16.358 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:16.394 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:16.571 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2020-10-09T17:04:16.673 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2020-10-09T17:04:20.509 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi184.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-10-09T17:04:20.720 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2020-10-09T17:04:20.821 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2020-10-09T17:07:02.761 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:06.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:11.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:15.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:20.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:24.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:29.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:33.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:38.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:43.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:07:52.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:01.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:16.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:16.439 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-09T17:08:22.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:22.457 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-09T17:08:28.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:28.475 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-09T17:08:34.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:34.489 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-10-09T17:08:40.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:08:40.574 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-10-09T17:08:40.576 INFO:teuthology.orchestra.run.smithi060:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-10-09T17:08:40.931 INFO:teuthology.provision.fog.smithi060:Node is ready 2020-10-09T17:08:40.931 INFO:teuthology.orchestra.run.smithi060:> hostname 2020-10-09T17:08:40.978 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2020-10-09T17:08:40.979 INFO:teuthology.orchestra.run.smithi060:> grep smithi060.front.sepia.ceph.com /etc/hosts 2020-10-09T17:08:41.027 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2020-10-09T17:08:41.028 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-10-09T17:08:41.156 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hosts 2020-10-09T17:08:41.173 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/172.21.15.60/172.21.15.60/g' /etc/hosts 2020-10-09T17:08:41.229 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hostname 2020-10-09T17:08:41.284 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-10-09T17:08:41.339 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2020-10-09T17:08:41.339 DEBUG:teuthology.parallel:result is None 2020-10-09T17:09:07.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:09:07.230 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-10-09T17:09:07.231 INFO:teuthology.orchestra.run.smithi184:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-10-09T17:09:07.567 INFO:teuthology.provision.fog.smithi184:Node is ready 2020-10-09T17:09:07.567 INFO:teuthology.orchestra.run.smithi184:> hostname 2020-10-09T17:09:07.615 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2020-10-09T17:09:07.616 INFO:teuthology.orchestra.run.smithi184:> grep smithi184.front.sepia.ceph.com /etc/hosts 2020-10-09T17:09:07.664 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2020-10-09T17:09:07.664 INFO:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2020-10-09T17:09:07.808 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hosts 2020-10-09T17:09:07.823 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/172.21.15.184/172.21.15.184/g' /etc/hosts 2020-10-09T17:09:07.879 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i -e 's/smithi184.front.sepia.ceph.com/smithi184/g' /etc/hostname 2020-10-09T17:09:07.936 INFO:teuthology.orchestra.run.smithi184:> sudo hostname smithi184 2020-10-09T17:09:07.990 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2020-10-09T17:09:07.990 DEBUG:teuthology.parallel:result is None 2020-10-09T17:09:08.132 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2020-10-09T17:09:08.133 INFO:teuthology.lock.ops:New key found. Updating... 2020-10-09T17:09:08.149 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com 2020-10-09T17:09:08.149 INFO:teuthology.lock.ops:New key found. Updating... 2020-10-09T17:09:08.162 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2020-10-09T17:09:08.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-09T17:09:08.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:09:08.234 INFO:teuthology.orchestra.run.smithi060:> true 2020-10-09T17:09:08.435 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-10-09T17:09:08.482 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-10-09T17:09:08.483 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-10-09T17:09:08.530 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-10-09T17:09:08.531 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-09T17:09:08.531 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-10-09T17:09:08.531 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-10-09T17:09:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-09T17:09:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-10-09T17:09:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-09T17:09:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-09T17:09:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-09T17:09:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-09T17:09:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-10-09T17:09:08.534 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-10-09T17:09:08.536 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-10-09T17:09:08.552 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server 2020-10-09T17:09:08.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-09T17:09:08.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:09:08.622 INFO:teuthology.orchestra.run.smithi184:> true 2020-10-09T17:09:08.815 INFO:teuthology.orchestra.run.smithi184:> uname -m 2020-10-09T17:09:08.863 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2020-10-09T17:09:08.864 INFO:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2020-10-09T17:09:08.911 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2020-10-09T17:09:08.911 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="18.04" 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-09T17:09:08.912 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-09T17:09:08.913 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-09T17:09:08.913 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=bionic 2020-10-09T17:09:08.913 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=bionic 2020-10-09T17:09:08.913 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2020-10-09T17:09:08.935 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-10-09T17:09:08.966 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu4kGk/54ot5aGD+tcFihNNIt3JdKbyUrwEeAO8YPODX3borROwRbqu9f4GkuXMUoe0btPn12aEp2s7+XTF5QjLdHTjwmOesvI7Zfh9ARuIxeQQ8XSYgu/qMv3A7wOFksQXvewsWtyOKssPg2Z/x+Vs4C71xtGdV269XQVtcBpQcsz9c+TDlWimzPugfuJH886Gq4ChdO4nf/6MGiCXb9/aeQhwx/wJjA7FtSWQ3Nb+tULeXr4rP3xwBfJz0DU1HhE693RY/R2k8Azq5qYlE1JWQxJLUNAG0BuQ818OAHqwtQJfTtNUYu/Pq6ww6EOSmICAP6Prx0l4/nZ2HPUoNlT smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy6vRl3U+6PLXomh+38cJLLwiy+eOMCw9NaMZpGI3Ig4qmkE16SSYZ8GYnNgTRnPeY31WUo3FlRYUl0AH8kA+VQ3XhMAM6//LFjx9xf3IhC/9rfw9V4u4rTTTUkeBAxtXGC8CrtYL1q9izwV5UZgOLJU+HD8ALyeR4p34r1fsqrvOrQoGyzTkgPmH6a4BiqYbfsT7mBCZHzQTClAYjpJOeOKBqfwxlcFpRQCy/CRrEdyt2JhaOsXzwRV6KBA31OMjUzOEM8rzoGUNYFP5uJ5V42D37VKPC7mN5PRp3a2GZfOrzVkMWBbzjzOC3IAMYA0HS3F6gRutl7ESPwf1wb6u5 2020-10-09T17:09:08.966 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-10-09T17:09:09.012 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-10-09T17:09:09.022 INFO:teuthology.task.internal:Saving configuration 2020-10-09T17:09:09.038 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-10-09T17:09:09.139 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-10-09T17:09:09.157 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-10-09 17:04:03.272727', 'description': '/home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIL5WFaqwSKXMydH5WvK2q4HeqtG/S2PHS9i6EN0KlTN', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi060.front.sepia.ceph.com'} 2020-10-09T17:09:09.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-10-09 17:04:03.272747', 'description': '/home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504', 'locked_by': 'scheduled_aemerson@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOPk7nOvewapGFxxeiPwAvFOVORJC30bpjsc6ElbNLj0', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi184.front.sepia.ceph.com'} 2020-10-09T17:09:09.173 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-10-09T17:09:09.185 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-10-09T17:09:09.185 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-10-09T17:09:09.186 INFO:teuthology.run_tasks:Running task console_log... 2020-10-09T17:09:09.353 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1375ccc048>, signals=[15]) 2020-10-09T17:09:09.354 INFO:teuthology.run_tasks:Running task internal.connect... 2020-10-09T17:09:09.398 INFO:teuthology.task.internal:Opening connections... 2020-10-09T17:09:09.399 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:09:09.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:09:09.469 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:09:09.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:09:09.544 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-10-09T17:09:09.555 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-10-09T17:09:09.673 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-10-09T17:09:09.673 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-10-09T17:09:09.726 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-10-09T17:09:09.726 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-09T17:09:09.726 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-09T17:09:09.727 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-09T17:09:09.728 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-10-09T17:09:09.728 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-10-09T17:09:09.728 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-10-09T17:09:09.744 INFO:teuthology.orchestra.run.smithi184:> uname -m 2020-10-09T17:09:09.749 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2020-10-09T17:09:09.749 INFO:teuthology.orchestra.run.smithi184:> cat /etc/os-release 2020-10-09T17:09:09.795 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu" 2020-10-09T17:09:09.795 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="18.04" 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/" 2020-10-09T17:09:09.796 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-10-09T17:09:09.797 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-10-09T17:09:09.797 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-10-09T17:09:09.797 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=bionic 2020-10-09T17:09:09.797 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=bionic 2020-10-09T17:09:09.797 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2020-10-09T17:09:09.814 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-10-09T17:09:09.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-10-09T17:09:09.903 INFO:teuthology.task.internal:Checking for old test directory... 2020-10-09T17:09:09.904 INFO:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2020-10-09T17:09:09.906 INFO:teuthology.orchestra.run.smithi184:> test '!' -e /home/ubuntu/cephtest 2020-10-09T17:09:09.911 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-10-09T17:09:09.975 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-10-09T17:09:09.975 INFO:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2020-10-09T17:09:09.978 INFO:teuthology.orchestra.run.smithi184:> test -z $(ls -A /var/lib/ceph) 2020-10-09T17:09:09.997 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-10-09T17:09:10.046 INFO:teuthology.run_tasks:Running task kernel... 2020-10-09T17:09:10.061 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-10-09T17:09:10.062 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-10-09T17:09:10.062 INFO:teuthology.orchestra.run.smithi060:> uname -r 2020-10-09T17:09:10.068 INFO:teuthology.orchestra.run.smithi060.stdout:4.15.0-118-generic 2020-10-09T17:09:10.068 INFO:teuthology.task.kernel:Running kernel on smithi060: 4.15.0-118-generic 2020-10-09T17:09:10.068 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2020-10-09T17:09:10.181 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-10-09T17:09:10.322 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-10-09T17:09:10.463 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-09T17:09:10.542 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-10-09T17:09:10.601 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,355 kB] 2020-10-09T17:09:10.703 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [824 kB] 2020-10-09T17:09:10.719 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [269 kB] 2020-10-09T17:09:10.722 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,069 kB] 2020-10-09T17:09:10.734 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [962 kB] 2020-10-09T17:09:10.742 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [239 kB] 2020-10-09T17:09:10.815 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-10-09T17:09:10.956 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,681 kB] 2020-10-09T17:09:11.313 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,119 kB] 2020-10-09T17:09:11.365 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [360 kB] 2020-10-09T17:09:11.385 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,542 kB] 2020-10-09T17:09:11.425 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,672 kB] 2020-10-09T17:09:11.470 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [352 kB] 2020-10-09T17:09:12.351 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 11.7 MB in 2s (6,009 kB/s) 2020-10-09T17:09:14.051 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-10-09T17:09:14.099 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-09T17:09:14.181 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-10-09T17:09:14.447 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-10-09T17:09:14.448 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-10-09T17:09:14.539 INFO:teuthology.orchestra.run.smithi060.stderr:E: Unable to locate package linux-image-current-generic 2020-10-09T17:09:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-09T17:09:14.541 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-09T17:09:14.620 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-10-09T17:09:14.882 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-10-09T17:09:14.882 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-10-09T17:09:15.102 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (4.15.0.118.105). 2020-10-09T17:09:15.103 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic set to manually installed. 2020-10-09T17:09:15.103 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2020-10-09T17:09:15.104 INFO:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2020-10-09T17:09:15.119 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2020-10-09T17:09:15.119 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2020-10-09T17:09:15.119 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 16 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 4.15.0.118.105 2020-10-09T17:09:15.120 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-09T17:09:15.121 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-4.15.0-118-generic, linux-modules-extra-4.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-09T17:09:15.121 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2020-10-09T17:09:15.121 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2020-10-09T17:09:15.121 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2020-10-09T17:09:15.121 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2020-10-09T17:09:15.121 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-10-09T17:09:15.122 INFO:teuthology.orchestra.run.smithi184:> uname -r 2020-10-09T17:09:15.128 INFO:teuthology.orchestra.run.smithi184.stdout:4.15.0-118-generic 2020-10-09T17:09:15.129 INFO:teuthology.task.kernel:Running kernel on smithi184: 4.15.0-118-generic 2020-10-09T17:09:15.129 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get clean 2020-10-09T17:09:15.316 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get update 2020-10-09T17:09:15.455 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-10-09T17:09:15.579 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-09T17:09:15.660 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-10-09T17:09:15.714 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,355 kB] 2020-10-09T17:09:15.816 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [824 kB] 2020-10-09T17:09:15.829 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [269 kB] 2020-10-09T17:09:15.833 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [962 kB] 2020-10-09T17:09:15.842 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,069 kB] 2020-10-09T17:09:15.854 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [239 kB] 2020-10-09T17:09:15.916 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-10-09T17:09:16.025 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,681 kB] 2020-10-09T17:09:16.401 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,119 kB] 2020-10-09T17:09:16.435 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [360 kB] 2020-10-09T17:09:16.447 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,672 kB] 2020-10-09T17:09:16.519 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,542 kB] 2020-10-09T17:09:16.571 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [352 kB] 2020-10-09T17:09:17.375 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 11.7 MB in 2s (6,242 kB/s) 2020-10-09T17:09:18.721 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-10-09T17:09:18.766 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-10-09T17:09:18.842 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-10-09T17:09:19.077 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-10-09T17:09:19.077 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-10-09T17:09:19.154 INFO:teuthology.orchestra.run.smithi184.stderr:E: Unable to locate package linux-image-current-generic 2020-10-09T17:09:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-10-09T17:09:19.156 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-10-09T17:09:19.234 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-10-09T17:09:19.462 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-10-09T17:09:19.462 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-10-09T17:09:19.660 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic is already the newest version (4.15.0.118.105). 2020-10-09T17:09:19.660 INFO:teuthology.orchestra.run.smithi184.stdout:linux-image-generic set to manually installed. 2020-10-09T17:09:19.661 INFO:teuthology.orchestra.run.smithi184.stdout:0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded. 2020-10-09T17:09:19.662 INFO:teuthology.orchestra.run.smithi184:> dpkg -s linux-image-generic 2020-10-09T17:09:19.683 INFO:teuthology.orchestra.run.smithi184.stdout:Package: linux-image-generic 2020-10-09T17:09:19.683 INFO:teuthology.orchestra.run.smithi184.stdout:Status: install ok installed 2020-10-09T17:09:19.683 INFO:teuthology.orchestra.run.smithi184.stdout:Priority: optional 2020-10-09T17:09:19.683 INFO:teuthology.orchestra.run.smithi184.stdout:Section: kernel 2020-10-09T17:09:19.683 INFO:teuthology.orchestra.run.smithi184.stdout:Installed-Size: 16 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Maintainer: Ubuntu Kernel Team 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Architecture: amd64 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Source: linux-meta 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Version: 4.15.0.118.105 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Depends: linux-image-4.15.0-118-generic, linux-modules-extra-4.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2020-10-09T17:09:19.684 INFO:teuthology.orchestra.run.smithi184.stdout:Recommends: thermald 2020-10-09T17:09:19.685 INFO:teuthology.orchestra.run.smithi184.stdout:Description: Generic Linux kernel image 2020-10-09T17:09:19.685 INFO:teuthology.orchestra.run.smithi184.stdout: This package will always depend on the latest generic kernel image 2020-10-09T17:09:19.685 INFO:teuthology.orchestra.run.smithi184.stdout: available. 2020-10-09T17:09:19.685 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-10-09T17:09:19.685 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-10-09T17:09:19.686 INFO:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-10-09T17:09:19.705 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2020-10-09T17:09:19.726 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-10-09T17:09:19.727 INFO:teuthology.orchestra.run.smithi184:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-10-09T17:09:19.745 INFO:teuthology.orchestra.run.smithi184.stdout:ttyS1 2020-10-09T17:09:19.768 INFO:teuthology.run_tasks:Running task internal.base... 2020-10-09T17:09:19.775 INFO:teuthology.task.internal:Creating test directory... 2020-10-09T17:09:19.775 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-10-09T17:09:19.777 INFO:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-10-09T17:09:19.815 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-10-09T17:09:19.904 INFO:teuthology.run_tasks:Running task internal.archive... 2020-10-09T17:09:20.005 INFO:teuthology.task.internal:Creating archive directory... 2020-10-09T17:09:20.005 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-10-09T17:09:20.008 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-10-09T17:09:20.020 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-10-09T17:09:20.078 INFO:teuthology.task.internal:Enabling coredump saving... 2020-10-09T17:09:20.079 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-10-09T17:09:20.081 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-10-09T17:09:20.102 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-09T17:09:20.104 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-09T17:09:20.111 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-09T17:09:20.113 INFO:teuthology.orchestra.run.smithi184.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-10-09T17:09:20.114 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-10-09T17:09:20.125 INFO:teuthology.task.internal:Configuring sudo... 2020-10-09T17:09:20.125 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-10-09T17:09:20.158 INFO:teuthology.orchestra.run.smithi184:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-10-09T17:09:20.174 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-10-09T17:09:20.184 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-10-09T17:09:20.184 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-10-09T17:09:20.214 INFO:teuthology.orchestra.run.smithi184:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-10-09T17:09:20.224 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-10-09T17:09:20.266 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-10-09T17:09:20.315 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:09:20.316 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-09T17:09:20.361 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-09T17:09:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:09:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:09:20.371 INFO:teuthology.orchestra.run.smithi060.stderr:152 bytes copied, 3.248e-05 s, 4.7 MB/s 2020-10-09T17:09:20.372 INFO:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-10-09T17:09:20.379 INFO:teuthology.orchestra.run.smithi184:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-10-09T17:09:20.428 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:09:20.429 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-09T17:09:20.474 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2020-10-09T17:09:20.507 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:09:20.507 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:09:20.508 INFO:teuthology.orchestra.run.smithi184.stderr:152 bytes copied, 3.3037e-05 s, 4.6 MB/s 2020-10-09T17:09:20.509 INFO:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2020-10-09T17:09:20.511 INFO:teuthology.orchestra.run.smithi184:> sudo service rsyslog restart 2020-10-09T17:09:20.591 INFO:teuthology.run_tasks:Running task internal.timer... 2020-10-09T17:09:20.599 INFO:teuthology.task.internal:Starting timer... 2020-10-09T17:09:20.599 INFO:teuthology.run_tasks:Running task pcp... 2020-10-09T17:09:20.676 INFO:teuthology.run_tasks:Running task selinux... 2020-10-09T17:09:20.794 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2020-10-09T17:09:20.795 DEBUG:teuthology.task.selinux:Excluding smithi184: OS 'ubuntu' does not support SELinux 2020-10-09T17:09:20.795 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-10-09T17:09:20.795 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-10-09T17:09:20.795 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-10-09T17:09:20.796 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-10-09T17:09:20.832 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-10-09T17:09:20.839 INFO:teuthology.repo_utils:Fetching master from origin 2020-10-09T17:09:20.898 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-10-09T17:09:20.910 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-10-09T17:09:20.912 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi184.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-10-09T17:09:21.233 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-10-09T17:09:22.118 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-10-09T17:09:24.149 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-10-09T17:09:24.171 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:24 +0000 (0:00:00.069) 0:00:00.069 ******** 2020-10-09T17:09:24.186 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:24 +0000 (0:00:00.014) 0:00:00.083 ******** 2020-10-09T17:09:25.841 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-10-09T17:09:25.885 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] 2020-10-09T17:09:25.923 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:25 +0000 (0:00:01.737) 0:00:01.820 ******** 2020-10-09T17:09:25.985 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:25 +0000 (0:00:00.061) 0:00:01.882 ******** 2020-10-09T17:09:26.281 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-10-09T17:09:26.283 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-09T17:09:26.316 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:26 +0000 (0:00:00.331) 0:00:02.213 ******** 2020-10-09T17:09:26.347 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:26 +0000 (0:00:00.031) 0:00:02.244 ******** 2020-10-09T17:09:26.859 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-10-09T17:09:26.861 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-10-09T17:09:26.914 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:26 +0000 (0:00:00.566) 0:00:02.811 ******** 2020-10-09T17:09:26.924 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-10-09T17:09:26.959 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:26 +0000 (0:00:00.045) 0:00:02.856 ******** 2020-10-09T17:09:27.332 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-10-09T17:09:27.333 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.069893", "end": "2020-10-09 17:09:27.308753", "rc": 0, "start": "2020-10-09 17:09:27.238860", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-10-09T17:09:27.364 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:27 +0000 (0:00:00.405) 0:00:03.262 ******** 2020-10-09T17:09:27.371 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.075245", "end": "2020-10-09 17:09:27.511410", "rc": 0, "start": "2020-10-09 17:09:27.436165", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-10-09T17:09:27.406 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:27 +0000 (0:00:00.041) 0:00:03.303 ******** 2020-10-09T17:09:27.600 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-10-09T17:09:27.601 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:09:27.635 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:27 +0000 (0:00:00.228) 0:00:03.532 ******** 2020-10-09T17:09:27.666 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:27 +0000 (0:00:00.031) 0:00:03.563 ******** 2020-10-09T17:09:28.252 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-10-09T17:09:28.254 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602263367.6480298-211646598904685/source", "state": "file", "uid": 0} 2020-10-09T17:09:28.286 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:28 +0000 (0:00:00.620) 0:00:04.183 ******** 2020-10-09T17:09:28.293 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602263367.6801007-142224461663592/source", "state": "file", "uid": 0} 2020-10-09T17:09:28.328 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:28 +0000 (0:00:00.041) 0:00:04.225 ******** 2020-10-09T17:09:28.583 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-10-09T17:09:28.585 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-10-09T17:09:28.586 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-10-09T17:09:28.617 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-10-09T17:09:28.619 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-10-09T17:09:28.730 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-10-09T17:09:28.738 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-10-09T17:09:28.887 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:28 +0000 (0:00:00.559) 0:00:04.785 ******** 2020-10-09T17:09:28.922 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:28 +0000 (0:00:00.034) 0:00:04.819 ******** 2020-10-09T17:09:29.635 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-10-09T17:09:29.679 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] 2020-10-09T17:09:29.721 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-10-09T17:09:29.723 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:29 +0000 (0:00:00.799) 0:00:05.618 ******** 2020-10-09T17:09:29.789 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:29 +0000 (0:00:00.067) 0:00:05.686 ******** 2020-10-09T17:09:30.078 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-10-09T17:09:30.080 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.173602", "end": "2020-10-09 17:09:30.054022", "rc": 0, "start": "2020-10-09 17:09:29.880420", "stderr": "", "stderr_lines": [], "stdout": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "stdout_lines": ["cb3b57134780b85e843ecc462024cfc477f6bcf1"]} 2020-10-09T17:09:30.112 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.323) 0:00:06.010 ******** 2020-10-09T17:09:30.130 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.155896", "end": "2020-10-09 17:09:30.104690", "rc": 0, "start": "2020-10-09 17:09:29.948794", "stderr": "", "stderr_lines": [], "stdout": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "stdout_lines": ["cb3b57134780b85e843ecc462024cfc477f6bcf1"]} 2020-10-09T17:09:30.164 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.051) 0:00:06.061 ******** 2020-10-09T17:09:30.252 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-10-09T17:09:30.254 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014595", "end": "2020-10-09 17:09:30.234897", "failed_when_result": false, "rc": 0, "start": "2020-10-09 17:09:30.220302", "stderr": "", "stderr_lines": [], "stdout": "9443d15ff7024b0631afa66ceb58a9a8ee18727c", "stdout_lines": ["9443d15ff7024b0631afa66ceb58a9a8ee18727c"]} 2020-10-09T17:09:30.287 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.123) 0:00:06.185 ******** 2020-10-09T17:09:30.305 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.019321", "end": "2020-10-09 17:09:30.481102", "failed_when_result": false, "rc": 0, "start": "2020-10-09 17:09:30.461781", "stderr": "", "stderr_lines": [], "stdout": "9443d15ff7024b0631afa66ceb58a9a8ee18727c", "stdout_lines": ["9443d15ff7024b0631afa66ceb58a9a8ee18727c"]} 2020-10-09T17:09:30.340 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.052) 0:00:06.237 ******** 2020-10-09T17:09:30.346 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-10-09T17:09:30.348 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.380 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.040) 0:00:06.278 ******** 2020-10-09T17:09:30.388 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.422 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.319 ******** 2020-10-09T17:09:30.428 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-10-09T17:09:30.430 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.464 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.361 ******** 2020-10-09T17:09:30.470 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.505 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.402 ******** 2020-10-09T17:09:30.511 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-10-09T17:09:30.513 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.546 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.444 ******** 2020-10-09T17:09:30.553 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.588 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.485 ******** 2020-10-09T17:09:30.594 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-10-09T17:09:30.596 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.629 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.041) 0:00:06.526 ******** 2020-10-09T17:09:30.636 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.672 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.042) 0:00:06.569 ******** 2020-10-09T17:09:30.679 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-10-09T17:09:30.680 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.716 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.043) 0:00:06.613 ******** 2020-10-09T17:09:30.722 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:30.758 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.042) 0:00:06.656 ******** 2020-10-09T17:09:30.848 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-10-09T17:09:30.850 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:09:30.883 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.124) 0:00:06.780 ******** 2020-10-09T17:09:30.891 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:09:30.927 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:30 +0000 (0:00:00.043) 0:00:06.824 ******** 2020-10-09T17:09:31.057 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-09T17:09:31.059 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-10-09T17:09:31.100 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-10-09T17:09:31.194 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-10-09T17:09:31.227 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-10-09T17:09:31.321 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-10-09T17:09:31.353 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-10-09T17:09:31.451 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-10-09T17:09:31.472 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-10-09T17:09:31.576 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-10-09T17:09:31.598 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-10-09T17:09:31.702 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-10-09T17:09:31.724 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-10-09T17:09:31.840 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-10-09T17:09:31.851 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-10-09T17:09:31.967 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-10-09T17:09:32.001 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-10-09T17:09:32.095 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-10-09T17:09:32.127 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-10-09T17:09:32.222 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-10-09T17:09:32.265 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-10-09T17:09:32.359 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-10-09T17:09:32.402 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-10-09T17:09:32.497 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-10-09T17:09:32.529 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-10-09T17:09:32.624 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-10-09T17:09:32.656 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-10-09T17:09:32.760 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-10-09T17:09:32.782 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-10-09T17:09:32.886 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-10-09T17:09:32.908 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-10-09T17:09:33.012 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-10-09T17:09:33.034 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-10-09T17:09:33.149 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-10-09T17:09:33.161 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-10-09T17:09:33.276 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-10-09T17:09:33.287 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-10-09T17:09:33.401 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-10-09T17:09:33.402 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-10-09T17:09:33.528 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-10-09T17:09:33.529 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-10-09T17:09:33.654 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-10-09T17:09:33.655 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-10-09T17:09:33.780 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-10-09T17:09:33.781 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-10-09T17:09:33.916 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-10-09T17:09:33.917 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-10-09T17:09:34.041 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-10-09T17:09:34.042 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-10-09T17:09:34.168 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-10-09T17:09:34.170 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-10-09T17:09:34.286 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-10-09T17:09:34.287 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-10-09T17:09:34.411 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-10-09T17:09:34.413 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-10-09T17:09:34.538 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-10-09T17:09:34.549 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-10-09T17:09:34.674 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-10-09T17:09:34.675 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-10-09T17:09:34.800 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-10-09T17:09:34.801 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-10-09T17:09:34.927 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-10-09T17:09:34.929 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-10-09T17:09:35.058 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-10-09T17:09:35.069 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-10-09T17:09:35.185 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-10-09T17:09:35.197 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-10-09T17:09:35.312 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-10-09T17:09:35.313 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-10-09T17:09:35.449 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-10-09T17:09:35.451 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-10-09T17:09:35.578 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-10-09T17:09:35.589 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-10-09T17:09:35.705 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-10-09T17:09:35.717 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-10-09T17:09:35.719 INFO:teuthology.task.ansible.out: 2020-10-09T17:09:35.835 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-10-09T17:09:35.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-10-09T17:09:35.974 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-10-09T17:09:35.976 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-10-09T17:09:36.100 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-09T17:09:36.101 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-09T17:09:36.227 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-10-09T17:09:36.238 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-10-09T17:09:36.354 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-10-09T17:09:36.365 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-10-09T17:09:36.491 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-10-09T17:09:36.492 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-10-09T17:09:36.618 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-10-09T17:09:36.619 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-10-09T17:09:36.736 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-10-09T17:09:36.749 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-10-09T17:09:36.865 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-10-09T17:09:36.876 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-10-09T17:09:37.003 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-10-09T17:09:37.015 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-10-09T17:09:37.142 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-09T17:09:37.154 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-09T17:09:37.279 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-10-09T17:09:37.290 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-10-09T17:09:37.415 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-10-09T17:09:37.417 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-10-09T17:09:37.541 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-10-09T17:09:37.553 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-10-09T17:09:37.678 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-10-09T17:09:37.689 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-10-09T17:09:37.814 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-10-09T17:09:37.815 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-10-09T17:09:37.940 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-10-09T17:09:37.942 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-10-09T17:09:38.070 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-10-09T17:09:38.072 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-10-09T17:09:38.197 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-10-09T17:09:38.198 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-10-09T17:09:38.323 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-10-09T17:09:38.324 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-10-09T17:09:38.449 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-10-09T17:09:38.460 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-10-09T17:09:38.586 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-10-09T17:09:38.588 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-10-09T17:09:38.713 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-10-09T17:09:38.714 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-10-09T17:09:38.840 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-10-09T17:09:38.842 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-10-09T17:09:38.966 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-10-09T17:09:38.979 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-10-09T17:09:39.104 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-10-09T17:09:39.104 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-10-09T17:09:39.230 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-10-09T17:09:39.231 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-10-09T17:09:39.367 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-09T17:09:39.378 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-10-09T17:09:39.493 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-10-09T17:09:39.506 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-10-09T17:09:39.623 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-10-09T17:09:39.634 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-10-09T17:09:39.738 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-10-09T17:09:39.760 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-10-09T17:09:39.875 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-10-09T17:09:39.887 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-10-09T17:09:39.992 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-10-09T17:09:40.024 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-10-09T17:09:40.128 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-10-09T17:09:40.150 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-10-09T17:09:40.245 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-10-09T17:09:40.277 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-10-09T17:09:40.382 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-10-09T17:09:40.404 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-10-09T17:09:40.500 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-10-09T17:09:40.544 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-10-09T17:09:40.627 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-10-09T17:09:40.669 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-10-09T17:09:40.763 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-10-09T17:09:40.805 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-10-09T17:09:40.889 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-10-09T17:09:40.933 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-10-09T17:09:41.017 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-10-09T17:09:41.060 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-10-09T17:09:41.155 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-10-09T17:09:41.199 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-10-09T17:09:41.281 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-10-09T17:09:41.333 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-10-09T17:09:41.408 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-10-09T17:09:41.461 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-10-09T17:09:41.535 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-10-09T17:09:41.587 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-10-09T17:09:41.661 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-10-09T17:09:41.726 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-10-09T17:09:41.790 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-10-09T17:09:41.843 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-10-09T17:09:41.918 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-10-09T17:09:41.982 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-10-09T17:09:42.056 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-10-09T17:09:42.109 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-10-09T17:09:42.194 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-10-09T17:09:42.248 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-10-09T17:09:42.322 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-10-09T17:09:42.375 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-10-09T17:09:42.450 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-10-09T17:09:42.494 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-10-09T17:09:42.579 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-10-09T17:09:42.623 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-10-09T17:09:42.708 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-10-09T17:09:42.761 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-10-09T17:09:42.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-10-09T17:09:42.889 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-10-09T17:09:42.973 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-10-09T17:09:43.026 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-10-09T17:09:43.089 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-10-09T17:09:43.153 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-10-09T17:09:43.218 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-10-09T17:09:43.281 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-10-09T17:09:43.345 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-10-09T17:09:43.412 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-10-09T17:09:43.475 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-10-09T17:09:43.549 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-10-09T17:09:43.613 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-10-09T17:09:43.676 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-10-09T17:09:43.739 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-10-09T17:09:43.813 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-10-09T17:09:43.866 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-10-09T17:09:43.940 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-10-09T17:09:43.994 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-10-09T17:09:44.068 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-10-09T17:09:44.122 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-10-09T17:09:44.207 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-10-09T17:09:44.262 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-09T17:09:44.347 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-10-09T17:09:44.389 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-10-09T17:09:44.485 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-10-09T17:09:44.529 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-10-09T17:09:44.623 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-10-09T17:09:44.669 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-10-09T17:09:44.753 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-10-09T17:09:44.795 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-10-09T17:09:44.890 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-10-09T17:09:44.922 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-10-09T17:09:45.027 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-10-09T17:09:45.059 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-10-09T17:09:45.163 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-10-09T17:09:45.185 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-10-09T17:09:45.290 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-10-09T17:09:45.302 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-10-09T17:09:45.423 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-10-09T17:09:45.435 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-10-09T17:09:45.562 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-10-09T17:09:45.573 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-10-09T17:09:45.699 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-10-09T17:09:45.699 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-10-09T17:09:45.824 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-10-09T17:09:45.836 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-10-09T17:09:45.950 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-10-09T17:09:45.962 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-10-09T17:09:46.089 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-10-09T17:09:46.090 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-10-09T17:09:46.216 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2020-10-09T17:09:46.227 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-10-09T17:09:46.343 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2020-10-09T17:09:46.344 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-10-09T17:09:46.472 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2020-10-09T17:09:46.474 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-10-09T17:09:46.603 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2020-10-09T17:09:46.625 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-10-09T17:09:46.740 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2020-10-09T17:09:46.752 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-10-09T17:09:46.867 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2020-10-09T17:09:46.889 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-10-09T17:09:46.994 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2020-10-09T17:09:47.017 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-10-09T17:09:47.133 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2020-10-09T17:09:47.144 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-10-09T17:09:47.261 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2020-10-09T17:09:47.272 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-10-09T17:09:47.388 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2020-10-09T17:09:47.410 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-10-09T17:09:47.515 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2020-10-09T17:09:47.558 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-10-09T17:09:47.663 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2020-10-09T17:09:47.687 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-10-09T17:09:47.783 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2020-10-09T17:09:47.816 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-10-09T17:09:47.925 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2020-10-09T17:09:48.075 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} Friday 09 October 2020 17:09:48 +0000 (0:00:17.136) 0:00:23.960 ******** 2020-10-09T17:09:48.093 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} TASK [users : Create all users without sudo access.] *************************** 2020-10-09T17:09:48.135 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:48 +0000 (0:00:00.071) 0:00:24.032 ******** 2020-10-09T17:09:48.176 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-10-09T17:09:48.219 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-10-09T17:09:48.221 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-10-09T17:09:48.222 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-10-09T17:09:48.224 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-10-09T17:09:48.225 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-10-09T17:09:48.227 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-10-09T17:09:48.228 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-10-09T17:09:48.230 INFO:teuthology.task.ansible.out: e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1 2020-10-09T17:09:48.230 INFO:teuthology.task.ansible.out:170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2d 2020-10-09T17:09:48.232 INFO:teuthology.task.ansible.out:f79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx 2020-10-09T17:09:48.233 INFO:teuthology.task.ansible.out:+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4 2020-10-09T17:09:48.235 INFO:teuthology.task.ansible.out:b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq 2020-10-09T17:09:48.236 INFO:teuthology.task.ansible.out:/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelop 2020-10-09T17:09:48.238 INFO:teuthology.task.ansible.out:ez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}]}, "changed": false} 2020-10-09T17:09:48.266 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:48 +0000 (0:00:00.130) 0:00:24.163 ******** 2020-10-09T17:09:48.303 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:48 +0000 (0:00:00.037) 0:00:24.200 ******** 2020-10-09T17:09:48.309 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-10-09T17:09:48.346 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:48 +0000 (0:00:00.042) 0:00:24.243 ******** 2020-10-09T17:09:48.348 INFO:teuthology.task.ansible.out: 2020-10-09T17:09:48.408 INFO:teuthology.task.ansible.out: 2020-10-09T17:09:48.410 INFO:teuthology.task.ansible.out:TASK [users : Merge managed_users and managed_admin users] ********************* 2020-10-09T17:09:48.413 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-10-09T17:09:48.414 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b26 2020-10-09T17:09:48.416 INFO:teuthology.task.ansible.out:5c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivanci 2020-10-09T17:09:48.418 INFO:teuthology.task.ansible.out:ch", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0Wihhvk 2020-10-09T17:09:48.420 INFO:teuthology.task.ansible.out:KztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2020-10-09T17:09:48.421 INFO:teuthology.task.ansible.out: HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2020-10-09T17:09:48.423 INFO:teuthology.task.ansible.out: e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2d 2020-10-09T17:09:48.425 INFO:teuthology.task.ansible.out:f79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": " 2020-10-09T17:09:48.425 INFO:teuthology.task.ansible.out:pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx 2020-10-09T17:09:48.427 INFO:teuthology.task.ansible.out:+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4 2020-10-09T17:09:48.429 INFO:teuthology.task.ansible.out:b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq 2020-10-09T17:09:48.430 INFO:teuthology.task.ansible.out:/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelop 2020-10-09T17:09:48.432 INFO:teuthology.task.ansible.out:ez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}]}, "changed": false} 2020-10-09T17:09:48.479 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:48 +0000 (0:00:00.132) 0:00:24.376 ******** 2020-10-09T17:09:49.265 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-10-09T17:09:49.267 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com -> localhost] => {"after": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "attempts": 1, "before": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "changed": false, "remote_url_changed": false} 2020-10-09T17:09:49.301 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:49 +0000 (0:00:00.822) 0:00:25.198 ******** 2020-10-09T17:09:49.308 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com -> localhost] => {"after": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "attempts": 1, "before": "cb3b57134780b85e843ecc462024cfc477f6bcf1", "changed": false, "remote_url_changed": false} 2020-10-09T17:09:49.343 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:09:49 +0000 (0:00:00.042) 0:00:25.240 ******** 2020-10-09T17:09:49.487 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-09T17:09:49.489 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-10-09T17:09:49.625 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-10-09T17:09:49.626 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-10-09T17:09:49.764 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-10-09T17:09:49.765 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-10-09T17:09:49.901 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-10-09T17:09:49.902 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-10-09T17:09:49.903 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-10-09T17:09:50.028 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-10-09T17:09:50.030 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-10-09T17:09:50.039 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-10-09T17:09:50.165 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-10-09T17:09:50.167 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-10-09T17:09:50.169 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-10-09T17:09:50.292 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-10-09T17:09:50.293 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-10-09T17:09:50.295 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-10-09T17:09:50.418 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-10-09T17:09:50.420 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-10-09T17:09:50.544 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-10-09T17:09:50.556 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-10-09T17:09:50.567 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:50.682 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-10-09T17:09:50.683 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:50.694 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-10-09T17:09:50.696 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-10-09T17:09:50.809 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-10-09T17:09:50.811 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-10-09T17:09:50.820 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-10-09T17:09:50.822 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:50.832 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:50.948 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2020-10-09T17:09:50.949 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:50.959 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-10-09T17:09:50.961 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:09:51.086 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-10-09T17:09:51.087 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9x 2020-10-09T17:09:51.089 INFO:teuthology.task.ansible.out:OZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-10-09T17:09:51.211 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-10-09T17:09:51.213 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSB 2020-10-09T17:09:51.215 INFO:teuthology.task.ansible.out:w9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-10-09T17:09:51.338 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-10-09T17:09:51.339 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-10-09T17:09:51.465 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-10-09T17:09:51.477 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-10-09T17:09:51.592 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-10-09T17:09:51.614 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-10-09T17:09:51.741 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-10-09T17:09:51.856 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-10-09T17:09:51.919 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-10-09T17:09:51.982 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-10-09T17:09:51.984 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-10-09T17:09:52.046 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-10-09T17:09:52.130 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-10-09T17:09:52.132 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-10-09T17:09:52.214 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-10-09T17:09:52.256 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-10-09T17:09:52.371 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-10-09T17:09:52.373 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-10-09T17:09:52.392 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-10-09T17:09:52.517 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-10-09T17:09:52.519 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-10-09T17:09:52.591 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-10-09T17:09:52.593 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-10-09T17:09:52.644 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-10-09T17:09:52.728 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-10-09T17:09:52.771 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-10-09T17:09:52.855 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-10-09T17:09:52.898 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-10-09T17:09:52.984 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-10-09T17:09:52.985 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-10-09T17:09:53.027 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-10-09T17:09:53.113 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-10-09T17:09:53.155 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-10-09T17:09:53.228 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-10-09T17:09:53.281 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-10-09T17:09:53.283 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-10-09T17:09:53.356 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-10-09T17:09:53.420 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-10-09T17:09:53.488 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-10-09T17:09:53.553 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-10-09T17:09:53.616 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-10-09T17:09:53.679 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-10-09T17:09:53.753 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-10-09T17:09:53.755 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-10-09T17:09:53.806 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-10-09T17:09:53.880 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-10-09T17:09:53.922 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-10-09T17:09:54.008 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-10-09T17:09:54.061 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-10-09T17:09:54.136 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-10-09T17:09:54.200 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-09T17:09:54.264 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-10-09T17:09:54.328 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-10-09T17:09:54.391 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-10-09T17:09:54.455 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-10-09T17:09:54.520 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-10-09T17:09:54.584 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-10-09T17:09:54.587 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-10-09T17:09:54.647 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-09T17:09:54.721 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-10-09T17:09:54.723 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-10-09T17:09:54.764 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-10-09T17:09:54.838 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-10-09T17:09:54.912 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-10-09T17:09:54.964 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-10-09T17:09:55.037 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-10-09T17:09:55.039 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-10-09T17:09:55.090 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-10-09T17:09:55.163 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-10-09T17:09:55.166 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-10-09T17:09:55.216 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-10-09T17:09:55.218 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-09T17:09:55.290 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-10-09T17:09:55.363 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-10-09T17:09:55.365 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-10-09T17:09:55.366 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-10-09T17:09:55.416 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-10-09T17:09:55.492 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-10-09T17:09:55.536 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-10-09T17:09:55.621 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-10-09T17:09:55.665 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-10-09T17:09:55.666 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-09T17:09:55.761 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-10-09T17:09:55.794 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-10-09T17:09:55.795 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-10-09T17:09:55.797 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-10-09T17:09:55.888 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-10-09T17:09:55.920 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-10-09T17:09:56.014 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-10-09T17:09:56.047 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-10-09T17:09:56.152 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-10-09T17:09:56.156 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-10-09T17:09:56.185 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-10-09T17:09:56.280 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-10-09T17:09:56.303 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-10-09T17:09:56.409 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-10-09T17:09:56.432 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-10-09T17:09:56.558 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-10-09T17:09:56.561 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "k 2020-10-09T17:09:56.562 INFO:teuthology.task.ansible.out:ey_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-10-09T17:09:56.696 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-10-09T17:09:56.708 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-10-09T17:09:56.826 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-10-09T17:09:56.847 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-10-09T17:09:56.952 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-10-09T17:09:56.954 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-10-09T17:09:56.975 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-10-09T17:09:56.976 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-10-09T17:09:57.084 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-10-09T17:09:57.095 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-10-09T17:09:57.210 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-10-09T17:09:57.232 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-10-09T17:09:57.350 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-10-09T17:09:57.352 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-10-09T17:09:57.360 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-10-09T17:09:57.477 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-10-09T17:09:57.479 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-09T17:09:57.606 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-10-09T17:09:57.608 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-10-09T17:09:57.734 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-10-09T17:09:57.745 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-10-09T17:09:57.863 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-10-09T17:09:57.875 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-10-09T17:09:57.991 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-10-09T17:09:58.013 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-10-09T17:09:58.118 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-10-09T17:09:58.140 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-10-09T17:09:58.245 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-10-09T17:09:58.288 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-10-09T17:09:58.372 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-10-09T17:09:58.405 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-10-09T17:09:58.491 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-10-09T17:09:58.533 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-10-09T17:09:58.618 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-10-09T17:09:58.671 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-10-09T17:09:58.746 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-10-09T17:09:58.799 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-10-09T17:09:58.874 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-10-09T17:09:58.927 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-10-09T17:09:58.991 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-10-09T17:09:59.054 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-10-09T17:09:59.117 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-10-09T17:09:59.181 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-10-09T17:09:59.236 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-10-09T17:09:59.300 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-10-09T17:09:59.364 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-10-09T17:09:59.427 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-10-09T17:09:59.490 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-10-09T17:09:59.557 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-10-09T17:09:59.610 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-10-09T17:09:59.683 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3d 2020-10-09T17:09:59.685 INFO:teuthology.task.ansible.out:UPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-10-09T17:09:59.747 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-10-09T17:09:59.801 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-10-09T17:09:59.875 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-10-09T17:09:59.928 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-10-09T17:10:00.002 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3d 2020-10-09T17:10:00.004 INFO:teuthology.task.ansible.out:UPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-10-09T17:10:00.055 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-10-09T17:10:00.129 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-10-09T17:10:00.181 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-10-09T17:10:00.245 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-10-09T17:10:00.308 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-10-09T17:10:00.310 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-10-09T17:10:00.371 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-10-09T17:10:00.444 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-10-09T17:10:00.446 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-10-09T17:10:00.497 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-10-09T17:10:00.560 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-10-09T17:10:00.633 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-10-09T17:10:00.635 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-10-09T17:10:00.686 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-10-09T17:10:00.759 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-10-09T17:10:00.761 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-10-09T17:10:00.812 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-10-09T17:10:00.886 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-10-09T17:10:00.939 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-10-09T17:10:01.012 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-10-09T17:10:01.065 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-10-09T17:10:01.128 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-10-09T17:10:01.193 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-10-09T17:10:01.258 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-10-09T17:10:01.321 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-10-09T17:10:01.384 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-10-09T17:10:01.447 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-10-09T17:10:01.449 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-10-09T17:10:01.511 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-10-09T17:10:01.575 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-10-09T17:10:01.638 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-10-09T17:10:01.702 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-10-09T17:10:01.766 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2020-10-09T17:10:01.768 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-10-09T17:10:01.830 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-10-09T17:10:01.893 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-10-09T17:10:01.956 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-10-09T17:10:02.009 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-10-09T17:10:02.085 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-10-09T17:10:02.138 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-10-09T17:10:02.212 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-10-09T17:10:02.265 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-10-09T17:10:02.340 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-10-09T17:10:02.342 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-09T17:10:02.393 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-10-09T17:10:02.456 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-10-09T17:10:02.519 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-10-09T17:10:02.593 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-10-09T17:10:02.647 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-10-09T17:10:02.649 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-10-09T17:10:02.724 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-10-09T17:10:02.776 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-10-09T17:10:02.840 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-10-09T17:10:02.904 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-10-09T17:10:02.989 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-10-09T17:10:03.032 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-10-09T17:10:03.107 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-10-09T17:10:03.161 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-10-09T17:10:03.235 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-10-09T17:10:03.292 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-10-09T17:10:03.366 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-10-09T17:10:03.420 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-10-09T17:10:03.493 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-10-09T17:10:03.536 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-10-09T17:10:03.630 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-10-09T17:10:03.673 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-10-09T17:10:03.757 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-10-09T17:10:03.801 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-10-09T17:10:03.885 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-10-09T17:10:03.917 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-10-09T17:10:04.012 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-10-09T17:10:04.044 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-10-09T17:10:04.139 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-10-09T17:10:04.171 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-10-09T17:10:04.265 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-10-09T17:10:04.296 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-10-09T17:10:04.391 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-10-09T17:10:04.423 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-10-09T17:10:04.518 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-10-09T17:10:04.551 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-10-09T17:10:04.635 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-10-09T17:10:04.678 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-10-09T17:10:04.773 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-10-09T17:10:04.775 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-10-09T17:10:04.806 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2020-10-09T17:10:04.900 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-10-09T17:10:04.933 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2020-10-09T17:10:05.020 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-10-09T17:10:05.062 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2020-10-09T17:10:05.065 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2020-10-09T17:10:05.148 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-10-09T17:10:05.190 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2020-10-09T17:10:05.273 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-10-09T17:10:05.316 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2020-10-09T17:10:05.401 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-10-09T17:10:05.444 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2020-10-09T17:10:05.529 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2020-10-09T17:10:05.583 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2020-10-09T17:10:05.658 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2020-10-09T17:10:05.713 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2020-10-09T17:10:05.776 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2020-10-09T17:10:05.829 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2020-10-09T17:10:05.916 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCavoAU8IGVih0HbUrIr5K44uQLdAq3r9MhKqnu754H8sigfFkxWPC6CsyKaGe0WnGcXz8wUdONqqew4WVvMN6KqE/8oLBf10yQYiigPw6ITPajrpj1JSsoW8kNIsfvrt+lzTUMWx0b6ThtQdANAu4OFZzERU//6yCy2Az0O0Ys9p4smgkwaXOA26hA48cxi147fBVC0Wl5xHuELKbmZ7pZFF/eAB9RLNlIRhBitWo93cLaWojivaJoxVhmKFgybuy7K+w1Tg7ACEJsSKXesO9AkcbFOXBuEwJQZDwyD4R6FfchloOGLBgb6Vn5bLDShWAgKlHE7eWa1EYG8qAjT9yeJEWIcQLz+8e3izFwriJAAzOqEB9a2GqVN6LQJzXTDcchJkOhhjfXaK3U3fqEfBozRtKVAUoNUgnlhYRbEcTMEuvz5AAUFB+0kvQpiw+RtieZUHl76KsNmFSIu7HSdgukdlkrGwOsl93qmYX96Q5AvanaAFj8ohbWSJSxZhKIB4U= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-10-09T17:10:05.990 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:05 +0000 (0:00:16.646) 0:00:41.887 ******** 2020-10-09T17:10:05.999 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2020-10-09T17:10:06.044 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-09T17:10:06.045 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.055 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.066 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.068 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.078 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.089 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.090 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2020-10-09T17:10:06.101 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-09T17:10:06.102 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.113 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.217 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-10-09T17:10:06.219 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCavoAU8IGVih0HbUrIr5K44uQLdAq3r9MhKqnu754H8sigfFkxWPC6CsyKaGe0WnGcXz8wUdONqqew4WVvMN6KqE/8oLBf10yQYiigPw6ITPajrpj1JSsoW8kNIsfvrt+lzTUMWx0b6ThtQdANAu4OFZzERU//6yCy2Az0O0Ys9p4smgkwaXOA26hA48cxi147fBVC0Wl5xHuELKbmZ7pZFF/eAB9RLNlIRhBitWo93cLaWojivaJoxVhmKFgybuy7K+w1Tg7ACEJsSKXesO9AkcbFOXBuEwJQZDwyD4R6FfchloOGLBgb6Vn5bLDShWAgKlHE7eWa1EYG8qAjT9yeJEWIcQLz+8e3izFwriJAAzOqEB9a2GqVN6LQJzXTDcchJkOhhjfXaK3U3fqEfBozRtKVAUoNUgnlhYRbEcTMEuvz5AAUFB+0kvQpiw+RtieZUHl76KsNmFSIu7HSdgukdlkrGwOsl93qmYX96Q5AvanaAFj8ohbWSJSxZhKIB4U= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2020-10-09T17:10:06.258 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-09T17:10:06.260 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-10-09T17:10:06.296 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:06 +0000 (0:00:00.306) 0:00:42.194 ******** 2020-10-09T17:10:06.304 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.306 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.340 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.342 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.351 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.362 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.364 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.374 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.385 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.387 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.397 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.398 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-10-09T17:10:06.524 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-10-09T17:10:06.525 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.536 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-10-09T17:10:06.538 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.548 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.549 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.559 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.561 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.571 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.582 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.584 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.593 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.605 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.615 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.617 INFO:teuthology.task.ansible.out: skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.627 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.639 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.640 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.651 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.661 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.663 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.673 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.684 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-10-09T17:10:06.686 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.696 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.697 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.708 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.719 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.720 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.740 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.742 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.752 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.763 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.774 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.775 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.785 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.796 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.798 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.809 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-10-09T17:10:06.810 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.811 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.822 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.824 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n' 2020-10-09T17:10:06.825 INFO:teuthology.task.ansible.out:}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.835 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.837 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.846 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.849 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.859 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.861 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.871 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.873 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.883 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.884 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.895 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.896 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.907 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.909 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.920 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.922 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.932 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.934 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.944 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.945 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.956 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.957 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:06.970 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.312 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@think 2020-10-09T17:10:07.314 INFO:teuthology.task.ansible.out:pad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXL 2020-10-09T17:10:07.315 INFO:teuthology.task.ansible.out:a7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33 2020-10-09T17:10:07.317 INFO:teuthology.task.ansible.out:bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansi 2020-10-09T17:10:07.319 INFO:teuthology.task.ansible.out:ble_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional res 2020-10-09T17:10:07.320 INFO:teuthology.task.ansible.out:ult was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed 2020-10-09T17:10:07.322 INFO:teuthology.task.ansible.out:": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec1 2020-10-09T17:10:07.323 INFO:teuthology.task.ansible.out:1bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJ 2020-10-09T17:10:07.325 INFO:teuthology.task.ansible.out:Ytm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.325 INFO:teuthology.task.ansible.out: 2020-10-09T17:10:07.326 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.327 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.328 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.329 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37 2020-10-09T17:10:07.330 INFO:teuthology.task.ansible.out:dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedor 2020-10-09T17:10:07.332 INFO:teuthology.task.ansible.out:a NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5 2020-10-09T17:10:07.333 INFO:teuthology.task.ansible.out:bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", " 2020-10-09T17:10:07.335 INFO:teuthology.task.ansible.out:changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.335 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.335 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-10-09T17:10:07.337 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.337 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a 2020-10-09T17:10:07.339 INFO:teuthology.task.ansible.out:5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.339 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.c 2020-10-09T17:10:07.340 INFO:teuthology.task.ansible.out:om] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.c 2020-10-09T17:10:07.342 INFO:teuthology.task.ansible.out:om ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477d 2020-10-09T17:10:07.343 INFO:teuthology.task.ansible.out:b6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Condition 2020-10-09T17:10:07.345 INFO:teuthology.task.ansible.out:al result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.345 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn" 2020-10-09T17:10:07.347 INFO:teuthology.task.ansible.out:: "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.347 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190 2020-10-09T17:10:07.348 INFO:teuthology.task.ansible.out:de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora aQ7SNya+XVgPs7WBN9jxwg 8a7b6c17443fb46062210cfded863de959fd3ac565fc86c759f1763364b6711d"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.349 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2 2020-10-09T17:10:07.350 INFO:teuthology.task.ansible.out:ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.360 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.361 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.372 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.374 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.384 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.386 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.401 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.403 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.439 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:07 +0000 (0:00:01.142) 0:00:43.336 ******** 2020-10-09T17:10:07.452 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.453 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.454 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.455 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.457 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.457 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.458 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn' 2020-10-09T17:10:07.460 INFO:teuthology.task.ansible.out:: 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.470 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.472 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.505 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:07 +0000 (0:00:00.066) 0:00:43.402 ******** 2020-10-09T17:10:07.514 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-10-09T17:10:07.516 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.517 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.518 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.518 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.529 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.540 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.542 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.551 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.553 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.563 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.574 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.575 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.590 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.601 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.603 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.612 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.623 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.625 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.634 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.636 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.647 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.658 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.660 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.669 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-09T17:10:07.671 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.682 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.693 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.695 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.705 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.706 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.716 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.717 INFO:teuthology.task.ansible.out: 2020-10-09T17:10:07.719 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.730 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.742 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.742 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.754 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain t8SWYptrljA5HM9+FH06Kg 535cbe1eb043fa4bd6c24ad0a43fa99375ee8c4dea15e10582e51c7a059f5b78"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.812 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:07 +0000 (0:00:00.306) 0:00:43.709 ******** 2020-10-09T17:10:07.830 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi060.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-10-09T17:10:07.842 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-10-09T17:10:07.843 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:07.875 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:07 +0000 (0:00:00.062) 0:00:43.772 ******** 2020-10-09T17:10:07.924 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-10-09T17:10:07.925 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-10-09T17:10:08.039 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-10-09T17:10:08.051 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-10-09T17:10:08.166 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-10-09T17:10:08.168 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-10-09T17:10:08.283 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-10-09T17:10:08.294 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-10-09T17:10:08.399 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-10-09T17:10:08.421 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-10-09T17:10:08.527 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-10-09T17:10:08.538 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-10-09T17:10:08.644 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-10-09T17:10:08.665 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-10-09T17:10:08.762 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-10-09T17:10:08.787 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-10-09T17:10:08.908 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-10-09T17:10:08.929 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-10-09T17:10:09.049 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-10-09T17:10:09.060 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-10-09T17:10:09.187 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-10-09T17:10:09.198 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-10-09T17:10:09.318 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-10-09T17:10:09.329 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-10-09T17:10:09.443 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-10-09T17:10:09.454 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-10-09T17:10:09.581 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-10-09T17:10:09.592 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-10-09T17:10:09.707 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-10-09T17:10:09.732 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-10-09T17:10:09.833 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-10-09T17:10:09.864 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-10-09T17:10:09.965 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-10-09T17:10:10.000 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-10-09T17:10:10.096 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-10-09T17:10:10.128 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-10-09T17:10:10.222 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-10-09T17:10:10.254 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-10-09T17:10:10.348 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-10-09T17:10:10.380 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-10-09T17:10:10.474 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-10-09T17:10:10.506 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-10-09T17:10:10.589 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-10-09T17:10:10.631 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-10-09T17:10:10.715 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-10-09T17:10:10.757 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-10-09T17:10:10.841 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-10-09T17:10:10.884 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-10-09T17:10:10.958 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-10-09T17:10:11.000 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-10-09T17:10:11.076 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-10-09T17:10:11.119 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-10-09T17:10:11.203 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-10-09T17:10:11.245 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-10-09T17:10:11.318 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-10-09T17:10:11.360 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-10-09T17:10:11.444 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-10-09T17:10:11.487 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-10-09T17:10:11.571 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-10-09T17:10:11.614 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-10-09T17:10:11.687 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-10-09T17:10:11.730 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-10-09T17:10:11.814 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-10-09T17:10:11.856 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-10-09T17:10:11.930 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-10-09T17:10:11.973 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-10-09T17:10:12.047 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-10-09T17:10:12.099 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-10-09T17:10:12.174 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-10-09T17:10:12.218 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-10-09T17:10:12.293 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-10-09T17:10:12.345 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-10-09T17:10:12.409 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-10-09T17:10:12.473 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-10-09T17:10:12.525 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-10-09T17:10:12.600 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-10-09T17:10:12.654 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-10-09T17:10:12.728 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-10-09T17:10:12.781 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-10-09T17:10:12.855 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-10-09T17:10:12.908 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-10-09T17:10:12.982 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-10-09T17:10:13.033 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:05.158) 0:00:48.930 ******** 2020-10-09T17:10:13.040 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-10-09T17:10:13.146 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-10-09T17:10:13.272 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-10-09T17:10:13.322 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.289) 0:00:49.220 ******** 2020-10-09T17:10:13.392 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-10-09T17:10:13.394 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "checksum": "cdbb99ea3d25e1bb5d121518ea51b3a3aaaa5795", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "074d9ab7c3dcfef391e763156bf7e463", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602263413.0536797-234214734089175/source", "state": "file", "uid": 0} 2020-10-09T17:10:13.437 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.114) 0:00:49.334 ******** 2020-10-09T17:10:13.465 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-10-09T17:10:13.466 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => { "msg": "Host smithi060.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-10-09T17:10:13.500 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.062) 0:00:49.397 ******** 2020-10-09T17:10:13.538 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-10-09T17:10:13.540 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-10-09T17:10:13.576 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.076) 0:00:49.473 ******** 2020-10-09T17:10:13.604 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-10-09T17:10:13.605 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:13.639 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.062) 0:00:49.536 ******** 2020-10-09T17:10:13.678 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-10-09T17:10:13.680 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "checksum": "cdbb99ea3d25e1bb5d121518ea51b3a3aaaa5795", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "074d9ab7c3dcfef391e763156bf7e463", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1602263413.344265-137795811212678/source", "state": "file", "uid": 0} 2020-10-09T17:10:13.713 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.074) 0:00:49.611 ******** 2020-10-09T17:10:13.746 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.032) 0:00:49.644 ******** 2020-10-09T17:10:13.753 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-10-09T17:10:13.755 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:13.790 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.043) 0:00:49.687 ******** 2020-10-09T17:10:13.797 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-10-09T17:10:13.798 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => { "msg": "Host smithi184.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-10-09T17:10:13.832 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.041) 0:00:49.729 ******** 2020-10-09T17:10:13.838 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-10-09T17:10:13.839 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:13.874 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.041) 0:00:49.771 ******** 2020-10-09T17:10:13.880 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-10-09T17:10:13.882 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-10-09T17:10:13.916 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.041) 0:00:49.813 ******** 2020-10-09T17:10:13.927 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-10-09T17:10:13.929 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:13.962 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:13 +0000 (0:00:00.045) 0:00:49.859 ******** 2020-10-09T17:10:13.969 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-10-09T17:10:13.971 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.005 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.043) 0:00:49.902 ******** 2020-10-09T17:10:14.012 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-10-09T17:10:14.013 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.047 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:49.944 ******** 2020-10-09T17:10:14.054 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-10-09T17:10:14.055 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.088 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:49.985 ******** 2020-10-09T17:10:14.094 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-10-09T17:10:14.096 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.130 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:50.027 ******** 2020-10-09T17:10:14.138 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-10-09T17:10:14.139 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.174 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.043) 0:00:50.071 ******** 2020-10-09T17:10:14.180 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.215 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:50.112 ******** 2020-10-09T17:10:14.222 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-10-09T17:10:14.224 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.257 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.042) 0:00:50.155 ******** 2020-10-09T17:10:14.263 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-10-09T17:10:14.265 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:14.299 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:50.196 ******** 2020-10-09T17:10:14.307 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-10-09T17:10:14.309 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.342 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.042) 0:00:50.239 ******** 2020-10-09T17:10:14.348 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-10-09T17:10:14.350 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.385 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.043) 0:00:50.282 ******** 2020-10-09T17:10:14.392 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-10-09T17:10:14.394 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.428 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.042) 0:00:50.325 ******** 2020-10-09T17:10:14.435 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-10-09T17:10:14.437 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.471 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.043) 0:00:50.368 ******** 2020-10-09T17:10:14.478 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-10-09T17:10:14.479 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.515 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.044) 0:00:50.412 ******** 2020-10-09T17:10:14.522 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-10-09T17:10:14.525 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:14.557 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:50.454 ******** 2020-10-09T17:10:14.565 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-10-09T17:10:14.567 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.601 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.044) 0:00:50.499 ******** 2020-10-09T17:10:14.610 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-10-09T17:10:14.611 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.651 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.049) 0:00:50.548 ******** 2020-10-09T17:10:14.659 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-10-09T17:10:14.661 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:14.699 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.048) 0:00:50.596 ******** 2020-10-09T17:10:14.707 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-10-09T17:10:14.709 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:14.743 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.044) 0:00:50.640 ******** 2020-10-09T17:10:14.750 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-10-09T17:10:14.752 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.788 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.044) 0:00:50.685 ******** 2020-10-09T17:10:14.796 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-10-09T17:10:14.798 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.832 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.043) 0:00:50.729 ******** 2020-10-09T17:10:14.839 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-10-09T17:10:14.841 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.874 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.041) 0:00:50.771 ******** 2020-10-09T17:10:14.884 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-10-09T17:10:14.885 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:14.919 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.045) 0:00:50.817 ******** 2020-10-09T17:10:14.927 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-10-09T17:10:14.929 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:14.964 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:14 +0000 (0:00:00.044) 0:00:50.861 ******** 2020-10-09T17:10:14.971 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-10-09T17:10:14.972 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.006 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.042) 0:00:50.903 ******** 2020-10-09T17:10:15.014 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-10-09T17:10:15.016 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.049 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.043) 0:00:50.947 ******** 2020-10-09T17:10:15.057 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-10-09T17:10:15.059 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.093 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.043) 0:00:50.990 ******** 2020-10-09T17:10:15.101 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-10-09T17:10:15.102 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:15.138 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.045) 0:00:51.036 ******** 2020-10-09T17:10:15.147 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-10-09T17:10:15.149 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:15.183 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.044) 0:00:51.080 ******** 2020-10-09T17:10:15.191 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-10-09T17:10:15.193 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.226 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.043) 0:00:51.123 ******** 2020-10-09T17:10:15.232 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-09T17:10:15.234 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.268 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.165 ******** 2020-10-09T17:10:15.275 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-10-09T17:10:15.277 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.310 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.042) 0:00:51.208 ******** 2020-10-09T17:10:15.317 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-10-09T17:10:15.319 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.352 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.249 ******** 2020-10-09T17:10:15.359 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-10-09T17:10:15.361 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.394 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.291 ******** 2020-10-09T17:10:15.400 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-10-09T17:10:15.402 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.435 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.333 ******** 2020-10-09T17:10:15.442 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-10-09T17:10:15.444 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.476 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.040) 0:00:51.374 ******** 2020-10-09T17:10:15.485 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-10-09T17:10:15.486 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.520 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.043) 0:00:51.417 ******** 2020-10-09T17:10:15.527 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-10-09T17:10:15.529 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:15.562 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.042) 0:00:51.459 ******** 2020-10-09T17:10:15.569 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-10-09T17:10:15.570 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.604 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.501 ******** 2020-10-09T17:10:15.611 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-09T17:10:15.612 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.657 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.053) 0:00:51.555 ******** 2020-10-09T17:10:15.664 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-10-09T17:10:15.666 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.699 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.042) 0:00:51.597 ******** 2020-10-09T17:10:15.706 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-10-09T17:10:15.708 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.744 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.044) 0:00:51.641 ******** 2020-10-09T17:10:15.750 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-10-09T17:10:15.751 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:15.785 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.683 ******** 2020-10-09T17:10:15.795 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-10-09T17:10:15.796 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.829 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.043) 0:00:51.726 ******** 2020-10-09T17:10:15.835 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-10-09T17:10:15.837 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.870 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.768 ******** 2020-10-09T17:10:15.876 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-10-09T17:10:15.878 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.911 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.040) 0:00:51.808 ******** 2020-10-09T17:10:15.917 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-10-09T17:10:15.919 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.953 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.041) 0:00:51.850 ******** 2020-10-09T17:10:15.959 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-10-09T17:10:15.960 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:15.993 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:15 +0000 (0:00:00.040) 0:00:51.890 ******** 2020-10-09T17:10:15.999 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-10-09T17:10:16.000 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.033 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.040) 0:00:51.931 ******** 2020-10-09T17:10:16.040 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-10-09T17:10:16.041 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.076 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:51.973 ******** 2020-10-09T17:10:16.082 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-10-09T17:10:16.084 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.118 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:52.015 ******** 2020-10-09T17:10:16.126 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-10-09T17:10:16.127 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:16.161 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:52.058 ******** 2020-10-09T17:10:16.167 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-10-09T17:10:16.169 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.203 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.041) 0:00:52.100 ******** 2020-10-09T17:10:16.210 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-10-09T17:10:16.212 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.245 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:52.142 ******** 2020-10-09T17:10:16.251 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-09T17:10:16.253 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.287 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.041) 0:00:52.184 ******** 2020-10-09T17:10:16.294 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-10-09T17:10:16.296 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.332 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.044) 0:00:52.229 ******** 2020-10-09T17:10:16.360 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-10-09T17:10:16.362 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.396 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.064) 0:00:52.294 ******** 2020-10-09T17:10:16.433 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-10-09T17:10:16.435 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.468 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.071) 0:00:52.365 ******** 2020-10-09T17:10:16.505 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-10-09T17:10:16.507 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.541 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.072) 0:00:52.438 ******** 2020-10-09T17:10:16.548 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.107349", "end": "2020-10-09 17:10:16.517658", "rc": 0, "start": "2020-10-09 17:10:16.410309", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-10-09T17:10:16.583 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.041) 0:00:52.480 ******** 2020-10-09T17:10:16.590 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-10-09T17:10:16.592 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.625 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:52.522 ******** 2020-10-09T17:10:16.632 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-10-09T17:10:16.633 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.670 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.044) 0:00:52.567 ******** 2020-10-09T17:10:16.699 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-10-09T17:10:16.701 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.734 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.064) 0:00:52.632 ******** 2020-10-09T17:10:16.762 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-09T17:10:16.764 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.796 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.061) 0:00:52.693 ******** 2020-10-09T17:10:16.824 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.858 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.061) 0:00:52.755 ******** 2020-10-09T17:10:16.865 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-09T17:10:16.867 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.080811", "end": "2020-10-09 17:10:17.012480", "rc": 0, "start": "2020-10-09 17:10:16.931669", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-10-09T17:10:16.899 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.041) 0:00:52.797 ******** 2020-10-09T17:10:16.906 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-10-09T17:10:16.908 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.942 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.042) 0:00:52.839 ******** 2020-10-09T17:10:16.948 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-10-09T17:10:16.950 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:16.984 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:16 +0000 (0:00:00.041) 0:00:52.881 ******** 2020-10-09T17:10:16.990 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-10-09T17:10:16.991 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.028 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.044) 0:00:52.925 ******** 2020-10-09T17:10:17.035 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-10-09T17:10:17.036 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.070 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.041) 0:00:52.967 ******** 2020-10-09T17:10:17.076 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-10-09T17:10:17.078 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.113 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.042) 0:00:53.010 ******** 2020-10-09T17:10:17.120 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-10-09T17:10:17.121 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.158 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.045) 0:00:53.055 ******** 2020-10-09T17:10:17.186 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-10-09T17:10:17.187 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.219 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.061) 0:00:53.116 ******** 2020-10-09T17:10:17.246 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-10-09T17:10:17.248 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.281 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.062) 0:00:53.179 ******** 2020-10-09T17:10:17.308 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-10-09T17:10:17.310 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.343 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.061) 0:00:53.241 ******** 2020-10-09T17:10:17.371 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-10-09T17:10:17.373 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.405 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.061) 0:00:53.302 ******** 2020-10-09T17:10:17.516 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-10-09T17:10:17.518 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.277783", "end": "2020-10-09 17:10:17.500335", "rc": 0, "start": "2020-10-09 17:10:17.222552", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:17.553 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.148) 0:00:53.451 ******** 2020-10-09T17:10:17.582 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-10-09T17:10:17.584 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:17.628 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.074) 0:00:53.525 ******** 2020-10-09T17:10:17.792 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-10-09T17:10:17.793 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.022373", "end": "2020-10-09 17:10:17.767076", "rc": 0, "start": "2020-10-09 17:10:17.744703", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:17.827 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:17 +0000 (0:00:00.198) 0:00:53.724 ******** 2020-10-09T17:10:18.327 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-10-09T17:10:18.329 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.797764", "end": "2020-10-09 17:10:18.500354", "rc": 0, "start": "2020-10-09 17:10:17.702590", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:18.367 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:18 +0000 (0:00:00.539) 0:00:54.264 ******** 2020-10-09T17:10:18.393 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-10-09T17:10:18.395 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:18.427 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:18 +0000 (0:00:00.060) 0:00:54.324 ******** 2020-10-09T17:10:18.569 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-10-09T17:10:18.572 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.014972", "end": "2020-10-09 17:10:18.747885", "rc": 0, "start": "2020-10-09 17:10:18.732913", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:18.605 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:18 +0000 (0:00:00.178) 0:00:54.503 ******** 2020-10-09T17:10:21.420 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1602263357, "cache_updated": false, "changed": false} 2020-10-09T17:10:21.453 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:21 +0000 (0:00:02.847) 0:00:57.350 ******** 2020-10-09T17:10:22.311 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263357, "cache_updated": false, "changed": false} 2020-10-09T17:10:22.348 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:22 +0000 (0:00:00.894) 0:00:58.245 ******** 2020-10-09T17:10:22.376 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-10-09T17:10:22.377 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:22.412 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:22 +0000 (0:00:00.064) 0:00:58.309 ******** 2020-10-09T17:10:22.514 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-10-09T17:10:22.515 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1602263420, "cache_updated": true, "changed": true} 2020-10-09T17:10:22.550 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:22 +0000 (0:00:00.138) 0:00:58.448 ******** 2020-10-09T17:10:22.977 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-10-09T17:10:22.979 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-10-09T17:10:23.012 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:23 +0000 (0:00:00.461) 0:00:58.909 ******** 2020-10-09T17:10:23.434 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-10-09T17:10:23.436 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:23.470 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:23 +0000 (0:00:00.458) 0:00:59.368 ******** 2020-10-09T17:10:23.498 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-10-09T17:10:23.499 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:23.533 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:23 +0000 (0:00:00.062) 0:00:59.430 ******** 2020-10-09T17:10:23.842 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-10-09T17:10:23.844 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-10-09T17:10:23.880 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:23 +0000 (0:00:00.347) 0:00:59.778 ******** 2020-10-09T17:10:23.887 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-10-09T17:10:23.889 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263357, "cache_updated": false, "changed": false} 2020-10-09T17:10:23.922 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:23 +0000 (0:00:00.041) 0:00:59.819 ******** 2020-10-09T17:10:24.271 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-09T17:10:24.273 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-10-09T17:10:24.552 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-10-09T17:10:24.759 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-10-09T17:10:24.760 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:24.794 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:24 +0000 (0:00:00.871) 0:01:00.691 ******** 2020-10-09T17:10:24.822 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-09T17:10:24.823 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-10-09T17:10:24.861 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:24 +0000 (0:00:00.066) 0:01:00.758 ******** 2020-10-09T17:10:24.992 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-10-09T17:10:24.994 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:10:25.027 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:25 +0000 (0:00:00.166) 0:01:00.925 ******** 2020-10-09T17:10:25.118 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-09T17:10:25.120 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-10-09T17:10:25.338 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-09T17:10:25.340 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-10-09T17:10:25.381 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-09T17:10:25.382 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-10-09T17:10:25.601 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-09T17:10:25.603 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-10-09T17:10:25.644 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-10-09T17:10:25.646 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-10-09T17:10:25.681 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:25 +0000 (0:00:00.653) 0:01:01.578 ******** 2020-10-09T17:10:25.806 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-10-09T17:10:25.808 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:10:25.841 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:25 +0000 (0:00:00.160) 0:01:01.739 ******** 2020-10-09T17:10:25.870 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-09T17:10:25.872 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-10-09T17:10:25.910 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:25 +0000 (0:00:00.068) 0:01:01.807 ******** 2020-10-09T17:10:25.950 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-10-09T17:10:25.952 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-10-09T17:10:25.986 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:25 +0000 (0:00:00.076) 0:01:01.884 ******** 2020-10-09T17:10:26.025 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-10-09T17:10:26.028 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.065 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.078) 0:01:01.963 ******** 2020-10-09T17:10:26.095 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-10-09T17:10:26.097 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:26.132 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.066) 0:01:02.029 ******** 2020-10-09T17:10:26.149 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-09T17:10:26.151 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-10-09T17:10:26.171 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-10-09T17:10:26.173 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.183 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.184 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.219 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.087) 0:01:02.116 ******** 2020-10-09T17:10:26.246 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-10-09T17:10:26.247 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.280 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.060) 0:01:02.177 ******** 2020-10-09T17:10:26.411 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-10-09T17:10:26.413 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-10-09T17:10:26.672 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-10-09T17:10:26.710 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.430) 0:01:02.607 ******** 2020-10-09T17:10:26.759 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-10-09T17:10:26.761 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-10-09T17:10:26.793 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.083) 0:01:02.691 ******** 2020-10-09T17:10:26.824 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-10-09T17:10:26.826 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.858 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.064) 0:01:02.755 ******** 2020-10-09T17:10:26.887 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-10-09T17:10:26.889 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:26.921 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:26 +0000 (0:00:00.063) 0:01:02.818 ******** 2020-10-09T17:10:26.970 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-10-09T17:10:26.971 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.973 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.983 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:26.995 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:27.029 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:27 +0000 (0:00:00.108) 0:01:02.927 ******** 2020-10-09T17:10:27.056 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-10-09T17:10:27.134 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 09 October 2020 17:10:27 +0000 (0:00:00.062) 0:01:02.989 ******** 2020-10-09T17:10:27.173 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-10-09T17:10:27.174 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263357, "cache_updated": false, "changed": false} 2020-10-09T17:10:27.209 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:27 +0000 (0:00:00.117) 0:01:03.106 ******** 2020-10-09T17:10:27.525 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-10-09T17:10:27.527 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-10-09T17:10:27.560 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:27 +0000 (0:00:00.350) 0:01:03.457 ******** 2020-10-09T17:10:27.876 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-10-09T17:10:27.878 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-10-09T17:10:27.911 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:27 +0000 (0:00:00.351) 0:01:03.808 ******** 2020-10-09T17:10:27.980 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-10-09T17:10:27.983 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:28.015 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.104) 0:01:03.912 ******** 2020-10-09T17:10:28.063 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-10-09T17:10:28.065 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-10-09T17:10:28.168 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-10-09T17:10:28.204 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.188) 0:01:04.101 ******** 2020-10-09T17:10:28.231 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-10-09T17:10:28.233 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:28.265 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.060) 0:01:04.162 ******** 2020-10-09T17:10:28.292 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-10-09T17:10:28.293 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:28.326 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.061) 0:01:04.223 ******** 2020-10-09T17:10:28.333 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-10-09T17:10:28.334 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-10-09T17:10:28.368 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.042) 0:01:04.266 ******** 2020-10-09T17:10:28.375 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-10-09T17:10:28.376 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:28.410 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.041) 0:01:04.307 ******** 2020-10-09T17:10:28.676 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-10-09T17:10:28.678 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-10-09T17:10:28.710 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.300) 0:01:04.608 ******** 2020-10-09T17:10:28.727 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-10-09T17:10:28.729 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-10-09T17:10:28.771 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:28 +0000 (0:00:00.059) 0:01:04.668 ******** 2020-10-09T17:10:28.852 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-10-09T17:10:28.920 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-10-09T17:10:28.968 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-10-09T17:10:29.005 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.234) 0:01:04.902 ******** 2020-10-09T17:10:29.033 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-10-09T17:10:29.035 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.068 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.062) 0:01:04.965 ******** 2020-10-09T17:10:29.096 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-10-09T17:10:29.098 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-10-09T17:10:29.130 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.062) 0:01:05.028 ******** 2020-10-09T17:10:29.159 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-10-09T17:10:29.160 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.195 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.064) 0:01:05.092 ******** 2020-10-09T17:10:29.381 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-10-09T17:10:29.383 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ActiveEnterTimestampMonotonic": "17843478", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target system.slice var-run.mount time-sync.target local-fs.target nss-lookup.target systemd-tmpfiles-setup.service systemd-journald.socket -.mount network.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:36 UTC", "AssertTimestampMonotonic": "17842493", "Before": "multi-user.target plymouth-quit.service shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ConditionTimestampMonotonic": "17842491", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-10-09T17:10:29.384 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1225", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ExecMainStartTimestampMonotonic": "17843439", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:36 UTC", "InactiveExitTimestampMonotonic": "17843478", "InvocationID": "027f1c7f8d314104b4fa7c8e78fb66f5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-10-09T17:10:29.386 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1225", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-10-09T17:10:29.387 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:36 UTC", "StateChangeTimestampMonotonic": "17843478", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:36 UTC", "WatchdogTimestampMonotonic": "17843475", "WatchdogUSec": "0"}} 2020-10-09T17:10:29.420 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.225) 0:01:05.318 ******** 2020-10-09T17:10:29.449 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.484 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.063) 0:01:05.381 ******** 2020-10-09T17:10:29.512 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.546 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.062) 0:01:05.443 ******** 2020-10-09T17:10:29.554 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-10-09T17:10:29.555 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2020-10-09T17:10:29.590 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.044) 0:01:05.488 ******** 2020-10-09T17:10:29.598 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-10-09T17:10:29.600 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.636 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.045) 0:01:05.533 ******** 2020-10-09T17:10:29.664 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-10-09T17:10:29.666 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.699 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.063) 0:01:05.596 ******** 2020-10-09T17:10:29.729 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-10-09T17:10:29.730 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.794 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.094) 0:01:05.691 ******** 2020-10-09T17:10:29.813 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-10-09T17:10:29.814 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:13 UTC", "ActiveEnterTimestampMonotonic": "18265943", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-journald.socket sysinit.target network.target basic.target var-run.mount -.mount remote-fs.target local-fs.target nss-lookup.target system.slice systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:13 UTC", "AssertTimestampMonotonic": "18265235", "Before": "multi-user.target shutdown.target plymouth-quit.service getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:13 UTC", "ConditionTimestampMonotonic": "18265235", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-10-09T17:10:29.816 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1507", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:13 UTC", "ExecMainStartTimestampMonotonic": "18265917", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:13 UTC", "InactiveExitTimestampMonotonic": "18265943", "InvocationID": "eea8bbb769ab49249be0b1cb163889b7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2020-10-09T17:10:29.817 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1507", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2020-10-09T17:10:29.822 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:13 UTC", "StateChangeTimestampMonotonic": "18265943", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:13 UTC", "WatchdogTimestampMonotonic": "18265942", "WatchdogUSec": "0"}} 2020-10-09T17:10:29.850 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.056) 0:01:05.748 ******** 2020-10-09T17:10:29.858 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-10-09T17:10:29.860 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.901 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.050) 0:01:05.798 ******** 2020-10-09T17:10:29.909 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-10-09T17:10:29.910 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:29.948 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:29 +0000 (0:00:00.046) 0:01:05.845 ******** 2020-10-09T17:10:29.956 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.005 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.056) 0:01:05.902 ******** 2020-10-09T17:10:30.013 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-10-09T17:10:30.015 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.052 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.045) 0:01:05.948 ******** 2020-10-09T17:10:30.058 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.097 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.046) 0:01:05.994 ******** 2020-10-09T17:10:30.103 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-10-09T17:10:30.106 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.143 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.046) 0:01:06.040 ******** 2020-10-09T17:10:30.150 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-10-09T17:10:30.152 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.185 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.042) 0:01:06.083 ******** 2020-10-09T17:10:30.188 INFO:teuthology.task.ansible.out: 2020-10-09T17:10:30.194 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-10-09T17:10:30.196 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.230 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.044) 0:01:06.127 ******** 2020-10-09T17:10:30.237 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-10-09T17:10:30.239 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.278 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.047) 0:01:06.174 ******** 2020-10-09T17:10:30.283 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-10-09T17:10:30.286 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.320 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.042) 0:01:06.217 ******** 2020-10-09T17:10:30.327 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-10-09T17:10:30.330 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-10-09T17:10:30.365 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.044) 0:01:06.262 ******** 2020-10-09T17:10:30.371 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-10-09T17:10:30.373 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.416 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.050) 0:01:06.313 ******** 2020-10-09T17:10:30.425 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-10-09T17:10:30.426 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-10-09T17:10:30.473 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.057) 0:01:06.371 ******** 2020-10-09T17:10:30.484 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.525 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.051) 0:01:06.422 ******** 2020-10-09T17:10:30.667 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-10-09T17:10:30.669 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-10-09T17:10:30.713 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.188) 0:01:06.611 ******** 2020-10-09T17:10:30.721 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-10-09T17:10:30.723 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.767 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.053) 0:01:06.664 ******** 2020-10-09T17:10:30.775 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-10-09T17:10:30.777 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-10-09T17:10:30.812 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.045) 0:01:06.709 ******** 2020-10-09T17:10:30.822 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.859 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.046) 0:01:06.756 ******** 2020-10-09T17:10:30.887 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-10-09T17:10:30.888 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:30.921 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.061) 0:01:06.818 ******** 2020-10-09T17:10:30.951 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-10-09T17:10:30.954 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} TASK [testnode : Include package type specific vars.] ************************** ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-10-09T17:10:30.987 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:30 +0000 (0:00:00.066) 0:01:06.884 ******** 2020-10-09T17:10:31.023 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.035) 0:01:06.920 ******** 2020-10-09T17:10:31.030 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-10-09T17:10:31.032 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-10-09T17:10:31.067 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.044) 0:01:06.964 ******** 2020-10-09T17:10:31.108 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-10-09T17:10:31.111 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-10-09T17:10:31.155 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.087) 0:01:07.052 ******** 2020-10-09T17:10:31.164 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-10-09T17:10:31.166 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "name": "ubuntu", "state": "present", "system": false} 2020-10-09T17:10:31.212 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.056) 0:01:07.108 ******** 2020-10-09T17:10:31.219 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-10-09T17:10:31.221 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-10-09T17:10:31.260 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.049) 0:01:07.158 ******** 2020-10-09T17:10:31.268 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-10-09T17:10:31.270 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:31.308 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.047) 0:01:07.205 ******** 2020-10-09T17:10:31.336 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-10-09T17:10:31.338 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:31.389 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.081) 0:01:07.286 ******** 2020-10-09T17:10:31.397 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-10-09T17:10:31.399 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-10-09T17:10:31.449 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.060) 0:01:07.347 ******** 2020-10-09T17:10:31.544 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-10-09T17:10:31.547 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-10-09T17:10:31.589 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.139) 0:01:07.487 ******** 2020-10-09T17:10:31.600 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-10-09T17:10:31.602 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "name": "ubuntu", "state": "present", "system": false} 2020-10-09T17:10:31.637 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.047) 0:01:07.534 ******** 2020-10-09T17:10:31.665 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:31.700 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.062) 0:01:07.597 ******** 2020-10-09T17:10:31.727 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-10-09T17:10:31.729 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:31.763 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.063) 0:01:07.661 ******** 2020-10-09T17:10:31.917 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-10-09T17:10:31.919 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-10-09T17:10:31.951 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:31 +0000 (0:00:00.187) 0:01:07.849 ******** 2020-10-09T17:10:31.970 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-10-09T17:10:31.972 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-10-09T17:10:32.006 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:32 +0000 (0:00:00.054) 0:01:07.903 ******** 2020-10-09T17:10:32.043 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-10-09T17:10:32.045 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi184"}, "changed": false} 2020-10-09T17:10:32.077 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:32 +0000 (0:00:00.071) 0:01:07.974 ******** 2020-10-09T17:10:32.325 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-10-09T17:10:32.362 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:32 +0000 (0:00:00.284) 0:01:08.259 ******** 2020-10-09T17:10:32.390 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-10-09T17:10:32.391 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi060"}, "changed": false} 2020-10-09T17:10:32.424 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:32 +0000 (0:00:00.062) 0:01:08.322 ******** 2020-10-09T17:10:32.951 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-10-09T17:10:32.953 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi060", "ansible_hostname": "smithi060", "ansible_nodename": "smithi060"}, "changed": false, "name": "smithi060"} 2020-10-09T17:10:32.985 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:32 +0000 (0:00:00.560) 0:01:08.883 ******** 2020-10-09T17:10:33.305 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-10-09T17:10:33.307 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-10-09T17:10:33.340 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:33 +0000 (0:00:00.354) 0:01:09.237 ******** 2020-10-09T17:10:33.547 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-10-09T17:10:33.549 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi184", "ansible_hostname": "smithi184", "ansible_nodename": "smithi184"}, "changed": false, "name": "smithi184"} 2020-10-09T17:10:33.589 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:33 +0000 (0:00:00.249) 0:01:09.486 ******** 2020-10-09T17:10:33.660 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-10-09T17:10:33.662 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-10-09T17:10:33.697 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:33 +0000 (0:00:00.107) 0:01:09.594 ******** 2020-10-09T17:10:33.910 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-10-09T17:10:33.912 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-10-09T17:10:33.947 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:33 +0000 (0:00:00.250) 0:01:09.844 ******** 2020-10-09T17:10:34.214 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-10-09T17:10:34.216 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-10-09T17:10:34.249 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.302) 0:01:10.147 ******** 2020-10-09T17:10:34.277 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-10-09T17:10:34.279 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.311 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.062) 0:01:10.209 ******** 2020-10-09T17:10:34.344 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.031) 0:01:10.240 ******** 2020-10-09T17:10:34.349 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-09T17:10:34.351 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.386 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.042) 0:01:10.283 ******** 2020-10-09T17:10:34.424 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-09T17:10:34.425 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.462 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.076) 0:01:10.360 ******** 2020-10-09T17:10:34.489 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-09T17:10:34.491 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.527 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.064) 0:01:10.424 ******** 2020-10-09T17:10:34.554 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-09T17:10:34.555 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.589 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.062) 0:01:10.486 ******** 2020-10-09T17:10:34.616 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-09T17:10:34.618 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.651 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.061) 0:01:10.548 ******** 2020-10-09T17:10:34.678 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-09T17:10:34.680 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.712 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.061) 0:01:10.609 ******** 2020-10-09T17:10:34.739 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-10-09T17:10:34.741 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.773 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.060) 0:01:10.670 ******** 2020-10-09T17:10:34.800 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-10-09T17:10:34.838 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.065) 0:01:10.736 ******** 2020-10-09T17:10:34.846 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-10-09T17:10:34.848 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-10-09T17:10:34.886 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.047) 0:01:10.783 ******** 2020-10-09T17:10:34.895 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-10-09T17:10:34.939 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.053) 0:01:10.836 ******** 2020-10-09T17:10:34.947 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-10-09T17:10:34.949 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:34.985 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:34 +0000 (0:00:00.045) 0:01:10.882 ******** 2020-10-09T17:10:34.991 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-10-09T17:10:35.027 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:10.925 ******** 2020-10-09T17:10:35.035 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.070 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:10.967 ******** 2020-10-09T17:10:35.076 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-10-09T17:10:35.078 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.111 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.041) 0:01:11.009 ******** 2020-10-09T17:10:35.119 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-09T17:10:35.120 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.175 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.063) 0:01:11.072 ******** 2020-10-09T17:10:35.185 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-10-09T17:10:35.186 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.186 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.221 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.046) 0:01:11.118 ******** 2020-10-09T17:10:35.229 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-09T17:10:35.230 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.263 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.161 ******** 2020-10-09T17:10:35.270 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-10-09T17:10:35.272 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.306 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.043) 0:01:11.204 ******** 2020-10-09T17:10:35.313 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-09T17:10:35.315 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.349 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.247 ******** 2020-10-09T17:10:35.355 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:10:35.357 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.391 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.041) 0:01:11.288 ******** 2020-10-09T17:10:35.398 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-09T17:10:35.400 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.433 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.041) 0:01:11.330 ******** 2020-10-09T17:10:35.440 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-10-09T17:10:35.441 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.476 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.373 ******** 2020-10-09T17:10:35.483 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-09T17:10:35.485 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.518 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.416 ******** 2020-10-09T17:10:35.526 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-10-09T17:10:35.528 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.563 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.044) 0:01:11.460 ******** 2020-10-09T17:10:35.570 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-10-09T17:10:35.572 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.606 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.503 ******** 2020-10-09T17:10:35.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-10-09T17:10:35.615 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.648 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.545 ******** 2020-10-09T17:10:35.657 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-10-09T17:10:35.693 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.045) 0:01:11.591 ******** 2020-10-09T17:10:35.700 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-10-09T17:10:35.702 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.736 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.633 ******** 2020-10-09T17:10:35.742 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-10-09T17:10:35.779 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.043) 0:01:11.677 ******** 2020-10-09T17:10:35.786 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-09T17:10:35.787 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.825 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.046) 0:01:11.723 ******** 2020-10-09T17:10:35.834 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-10-09T17:10:35.868 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.765 ******** 2020-10-09T17:10:35.875 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-09T17:10:35.876 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.911 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.042) 0:01:11.808 ******** 2020-10-09T17:10:35.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-10-09T17:10:35.918 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.954 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.043) 0:01:11.851 ******** 2020-10-09T17:10:35.961 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:10:35.963 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:35.999 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:35 +0000 (0:00:00.045) 0:01:11.897 ******** 2020-10-09T17:10:36.006 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-10-09T17:10:36.007 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.041 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.041) 0:01:11.938 ******** 2020-10-09T17:10:36.048 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-10-09T17:10:36.050 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.083 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.042) 0:01:11.981 ******** 2020-10-09T17:10:36.091 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-10-09T17:10:36.093 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.127 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.042) 0:01:12.024 ******** 2020-10-09T17:10:36.135 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-09T17:10:36.137 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.188 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.061) 0:01:12.085 ******** 2020-10-09T17:10:36.199 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:10:36.201 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.242 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.053) 0:01:12.139 ******** 2020-10-09T17:10:36.250 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-09T17:10:36.251 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.287 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.045) 0:01:12.185 ******** 2020-10-09T17:10:36.294 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-10-09T17:10:36.296 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.331 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.228 ******** 2020-10-09T17:10:36.337 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-10-09T17:10:36.339 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.375 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.272 ******** 2020-10-09T17:10:36.382 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-10-09T17:10:36.384 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.420 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.045) 0:01:12.318 ******** 2020-10-09T17:10:36.427 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-10-09T17:10:36.429 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.463 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.042) 0:01:12.360 ******** 2020-10-09T17:10:36.471 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-10-09T17:10:36.473 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.507 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.403 ******** 2020-10-09T17:10:36.513 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-10-09T17:10:36.514 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.557 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.050) 0:01:12.454 ******** 2020-10-09T17:10:36.564 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-10-09T17:10:36.566 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.601 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.498 ******** 2020-10-09T17:10:36.607 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-10-09T17:10:36.609 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.644 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.541 ******** 2020-10-09T17:10:36.651 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.690 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.045) 0:01:12.587 ******** 2020-10-09T17:10:36.697 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-10-09T17:10:36.699 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.734 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.044) 0:01:12.631 ******** 2020-10-09T17:10:36.742 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-09T17:10:36.744 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.778 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.044) 0:01:12.675 ******** 2020-10-09T17:10:36.787 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-10-09T17:10:36.789 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.823 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.045) 0:01:12.721 ******** 2020-10-09T17:10:36.831 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:10:36.833 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.868 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.044) 0:01:12.765 ******** 2020-10-09T17:10:36.874 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-10-09T17:10:36.876 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.911 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.043) 0:01:12.808 ******** 2020-10-09T17:10:36.918 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-10-09T17:10:36.919 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.952 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.041) 0:01:12.850 ******** 2020-10-09T17:10:36.959 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-10-09T17:10:36.960 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:36.994 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:36 +0000 (0:00:00.041) 0:01:12.892 ******** 2020-10-09T17:10:37.001 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-09T17:10:37.003 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.037 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.042) 0:01:12.934 ******** 2020-10-09T17:10:37.071 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-09T17:10:37.073 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.105 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.068) 0:01:13.002 ******** 2020-10-09T17:10:37.132 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-10-09T17:10:37.134 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for custom repos] *************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005223", "end": "2020-10-09 17:10:37.114905", "rc": 0, "start": "2020-10-09 17:10:37.109682", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:37.167 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.062) 0:01:13.064 ******** 2020-10-09T17:10:37.200 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.032) 0:01:13.097 ******** 2020-10-09T17:10:37.207 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-10-09T17:10:37.242 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.042) 0:01:13.139 ******** 2020-10-09T17:10:37.248 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-10-09T17:10:37.250 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.284 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.042) 0:01:13.182 ******** 2020-10-09T17:10:37.312 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-10-09T17:10:37.313 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.348 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.063) 0:01:13.246 ******** 2020-10-09T17:10:37.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2020-10-09T17:10:37.377 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.411 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.062) 0:01:13.308 ******** 2020-10-09T17:10:37.438 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-10-09T17:10:37.440 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.474 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.063) 0:01:13.371 ******** 2020-10-09T17:10:37.501 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-10-09T17:10:37.502 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.536 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.062) 0:01:13.433 ******** 2020-10-09T17:10:37.564 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-10-09T17:10:37.566 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set apt preferences] ****************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-10-09T17:10:37.600 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.064) 0:01:13.497 ******** 2020-10-09T17:10:37.634 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.033) 0:01:13.531 ******** 2020-10-09T17:10:37.641 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-10-09T17:10:37.642 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.679 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.045) 0:01:13.577 ******** 2020-10-09T17:10:37.686 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-10-09T17:10:37.688 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:37.726 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.046) 0:01:13.623 ******** 2020-10-09T17:10:37.871 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-10-09T17:10:37.873 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004827", "end": "2020-10-09 17:10:38.045364", "rc": 0, "start": "2020-10-09 17:10:38.040537", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:37.911 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.185) 0:01:13.808 ******** 2020-10-09T17:10:37.938 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-10-09T17:10:37.974 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:37 +0000 (0:00:00.062) 0:01:13.871 ******** 2020-10-09T17:10:38.303 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-10-09T17:10:38.304 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-10-09T17:10:38.337 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:38 +0000 (0:00:00.363) 0:01:14.235 ******** 2020-10-09T17:10:38.365 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-10-09T17:10:38.367 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:38.399 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:38 +0000 (0:00:00.061) 0:01:14.296 ******** 2020-10-09T17:10:38.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-10-09T17:10:38.984 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-10-09T17:10:39.390 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-10-09T17:10:39.940 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-10-09T17:10:39.975 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:39 +0000 (0:00:01.575) 0:01:15.872 ******** 2020-10-09T17:10:40.316 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-10-09T17:10:40.349 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:40 +0000 (0:00:00.374) 0:01:16.247 ******** 2020-10-09T17:10:40.919 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-10-09T17:10:40.920 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:40.956 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:40 +0000 (0:00:00.606) 0:01:16.854 ******** 2020-10-09T17:10:40.985 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-10-09T17:10:41.022 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:41 +0000 (0:00:00.065) 0:01:16.920 ******** 2020-10-09T17:10:41.186 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-10-09T17:10:41.188 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033726", "end": "2020-10-09 17:10:41.166154", "rc": 0, "start": "2020-10-09 17:10:41.132428", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:41.221 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:41 +0000 (0:00:00.198) 0:01:17.118 ******** 2020-10-09T17:10:41.248 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263357, "cache_updated": false, "changed": false} 2020-10-09T17:10:41.284 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:41 +0000 (0:00:00.063) 0:01:17.182 ******** 2020-10-09T17:10:41.312 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-10-09T17:10:41.347 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:41 +0000 (0:00:00.062) 0:01:17.244 ******** 2020-10-09T17:10:41.510 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-10-09T17:10:41.512 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034155", "end": "2020-10-09 17:10:41.684725", "rc": 0, "start": "2020-10-09 17:10:41.650570", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:10:41.545 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:41 +0000 (0:00:00.197) 0:01:17.442 ******** 2020-10-09T17:10:44.063 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-10-09T17:10:44.065 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:44.107 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:44 +0000 (0:00:02.561) 0:01:20.004 ******** 2020-10-09T17:10:45.718 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": true, "changed": true} 2020-10-09T17:10:45.768 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:45 +0000 (0:00:01.661) 0:01:21.665 ******** 2020-10-09T17:10:54.019 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-10-09T17:10:54.021 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102060 files and directories 2020-10-09T17:10:54.023 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-10-09T17:10:54.024 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102060 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-09T17:10:54.055 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:54 +0000 (0:00:08.286) 0:01:29.952 ******** 2020-10-09T17:10:54.979 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-09T17:10:54.981 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:10:55.014 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:55 +0000 (0:00:00.959) 0:01:30.911 ******** 2020-10-09T17:10:55.021 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-10-09T17:10:55.021 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102060 files and directories 2020-10-09T17:10:55.023 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 7 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-10-09T17:10:55.024 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102060 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-09T17:10:55.057 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:55 +0000 (0:00:00.043) 0:01:30.955 ******** 2020-10-09T17:10:55.064 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-10-09T17:10:55.065 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:55.100 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:55 +0000 (0:00:00.042) 0:01:30.997 ******** 2020-10-09T17:10:55.900 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-09T17:10:55.902 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:10:55.935 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:55 +0000 (0:00:00.834) 0:01:31.832 ******** 2020-10-09T17:10:55.962 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-10-09T17:10:55.964 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:10:55.998 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:10:55 +0000 (0:00:00.063) 0:01:31.895 ******** 2020-10-09T17:11:03.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:11:03.935 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (9661 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-10-09T17:11:03.936 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101918 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-10-09T17:11:03.938 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (9661 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-10-09T17:11:03.939 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101918 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-09T17:11:03.973 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:03 +0000 (0:00:07.974) 0:01:39.870 ******** 2020-10-09T17:11:04.862 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-10-09T17:11:04.864 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:11:04.898 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:04 +0000 (0:00:00.925) 0:01:40.796 ******** 2020-10-09T17:11:04.968 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:11:04.969 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]\nFetched 10.4 MB in 1s (9706 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-10-09T17:11:04.970 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101918 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-10-09T17:11:04.972 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.4 [2727 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.4 [924 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.32 [40.3 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.32 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.32 [5184 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.32 [869 kB]", "Fetched 10.4 MB in 1s (9706 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-10-09T17:11:04.973 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101918 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.4_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.32_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.4) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.32) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.32) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-10-09T17:11:05.005 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.106) 0:01:40.903 ******** 2020-10-09T17:11:05.022 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-10-09T17:11:05.024 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-10-09T17:11:05.058 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.052) 0:01:40.955 ******** 2020-10-09T17:11:05.086 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-10-09T17:11:05.088 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:05.122 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.064) 0:01:41.019 ******** 2020-10-09T17:11:05.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-10-09T17:11:05.288 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.165) 0:01:41.185 ******** 2020-10-09T17:11:05.617 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-10-09T17:11:05.619 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-10-09T17:11:05.651 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.363) 0:01:41.549 ******** 2020-10-09T17:11:05.804 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-10-09T17:11:05.805 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1230, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-10-09T17:11:05.838 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.186) 0:01:41.735 ******** 2020-10-09T17:11:05.908 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-10-09T17:11:05.909 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:11:05.946 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:05 +0000 (0:00:00.107) 0:01:41.843 ******** 2020-10-09T17:11:05.977 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-10-09T17:11:05.979 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:11:06.013 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.066) 0:01:41.910 ******** 2020-10-09T17:11:06.075 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-10-09T17:11:06.077 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-10-09T17:11:06.111 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.097) 0:01:42.008 ******** 2020-10-09T17:11:06.138 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-10-09T17:11:06.140 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.172 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.061) 0:01:42.069 ******** 2020-10-09T17:11:06.315 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-10-09T17:11:06.316 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-10-09T17:11:06.317 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-10-09T17:11:06.319 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:14 UTC", "ActiveEnterTimestampMonotonic": "18686053", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount nss-lookup.target systemd-journald.socket basic.target sysinit.target network.target systemd-tmpfiles-setup.service remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:13 UTC", "AssertTimestampMonotonic": "18264386", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:13 UTC", "ConditionTimestampMonotonic": "18264386", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-10-09T17:11:06.321 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1588", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:14 UTC", "ExecMainStartTimestampMonotonic": "18686042", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:13 UTC", "InactiveExitTimestampMonotonic": "18265194", "InvocationID": "7703051bd1dd446bb1ca6465baf5522a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-10-09T17:11:06.322 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1588", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-10-09T17:11:06.324 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:14 UTC", "StateChangeTimestampMonotonic": "18686053", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:14 UTC", "WatchdogTimestampMonotonic": "18686052", "WatchdogUSec": "0"}} 2020-10-09T17:11:06.352 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.180) 0:01:42.250 ******** 2020-10-09T17:11:06.394 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.041) 0:01:42.291 ******** 2020-10-09T17:11:06.400 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-09T17:11:06.402 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.437 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.043) 0:01:42.334 ******** 2020-10-09T17:11:06.472 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-09T17:11:06.474 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.506 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.068) 0:01:42.403 ******** 2020-10-09T17:11:06.533 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-09T17:11:06.535 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.570 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.063) 0:01:42.467 ******** 2020-10-09T17:11:06.599 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.632 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.061) 0:01:42.529 ******** 2020-10-09T17:11:06.659 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-09T17:11:06.661 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.693 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.060) 0:01:42.590 ******** 2020-10-09T17:11:06.722 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-10-09T17:11:06.724 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.757 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.063) 0:01:42.654 ******** 2020-10-09T17:11:06.789 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.032) 0:01:42.686 ******** 2020-10-09T17:11:06.797 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-09T17:11:06.799 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.832 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.042) 0:01:42.729 ******** 2020-10-09T17:11:06.861 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-09T17:11:06.862 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.895 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.062) 0:01:42.792 ******** 2020-10-09T17:11:06.922 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:06.924 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:06.956 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.061) 0:01:42.854 ******** 2020-10-09T17:11:06.963 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-10-09T17:11:06.965 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1230, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-10-09T17:11:06.998 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:06 +0000 (0:00:00.041) 0:01:42.896 ******** 2020-10-09T17:11:07.004 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-09T17:11:07.005 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.039 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.040) 0:01:42.936 ******** 2020-10-09T17:11:07.066 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:11:07.068 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.099 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.060) 0:01:42.997 ******** 2020-10-09T17:11:07.127 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-10-09T17:11:07.129 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.164 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.064) 0:01:43.061 ******** 2020-10-09T17:11:07.196 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.031) 0:01:43.093 ******** 2020-10-09T17:11:07.203 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-09T17:11:07.205 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.238 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.042) 0:01:43.135 ******** 2020-10-09T17:11:07.266 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-09T17:11:07.268 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.300 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.061) 0:01:43.197 ******** 2020-10-09T17:11:07.327 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:07.329 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.361 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.060) 0:01:43.258 ******** 2020-10-09T17:11:07.388 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-09T17:11:07.389 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.421 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.060) 0:01:43.319 ******** 2020-10-09T17:11:07.449 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-09T17:11:07.451 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.483 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.061) 0:01:43.380 ******** 2020-10-09T17:11:07.491 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-10-09T17:11:07.492 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ActiveEnterTimestampMonotonic": "18272915", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount nss-lookup.target basic.target sysinit.target network.target systemd-tmpfiles-setup.service system.slice systemd-journald.socket remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:36 UTC", "AssertTimestampMonotonic": "17840046", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ConditionTimestampMonotonic": "17840046", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-10-09T17:11:07.494 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1342", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ExecMainStartTimestampMonotonic": "18272904", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:36 UTC", "InactiveExitTimestampMonotonic": "17840961", "InvocationID": "de70c04bcf4544e49ce514f719a2eb72", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-10-09T17:11:07.495 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1342", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-10-09T17:11:07.497 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:36 UTC", "StateChangeTimestampMonotonic": "18272915", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:36 UTC", "WatchdogTimestampMonotonic": "18272914", "WatchdogUSec": "0"}} 2020-10-09T17:11:07.527 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.044) 0:01:43.424 ******** 2020-10-09T17:11:07.533 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-09T17:11:07.535 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.569 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.042) 0:01:43.467 ******** 2020-10-09T17:11:07.576 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-10-09T17:11:07.578 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.610 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.040) 0:01:43.507 ******** 2020-10-09T17:11:07.616 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-09T17:11:07.618 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.651 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.040) 0:01:43.548 ******** 2020-10-09T17:11:07.657 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-10-09T17:11:07.660 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.692 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.041) 0:01:43.589 ******** 2020-10-09T17:11:07.700 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.734 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.041) 0:01:43.631 ******** 2020-10-09T17:11:07.741 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-10-09T17:11:07.742 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.776 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.042) 0:01:43.674 ******** 2020-10-09T17:11:07.783 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:07.784 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.821 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.044) 0:01:43.719 ******** 2020-10-09T17:11:07.828 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-10-09T17:11:07.830 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.863 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.041) 0:01:43.760 ******** 2020-10-09T17:11:07.870 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-09T17:11:07.872 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.905 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.042) 0:01:43.802 ******** 2020-10-09T17:11:07.912 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-10-09T17:11:07.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.947 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.042) 0:01:43.844 ******** 2020-10-09T17:11:07.953 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-09T17:11:07.955 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:07.989 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:07 +0000 (0:00:00.041) 0:01:43.886 ******** 2020-10-09T17:11:07.996 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-10-09T17:11:07.997 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.030 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.041) 0:01:43.928 ******** 2020-10-09T17:11:08.037 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:08.038 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.075 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.044) 0:01:43.972 ******** 2020-10-09T17:11:08.082 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-10-09T17:11:08.083 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.117 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.041) 0:01:44.014 ******** 2020-10-09T17:11:08.123 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-09T17:11:08.124 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.158 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.041) 0:01:44.055 ******** 2020-10-09T17:11:08.167 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-10-09T17:11:08.168 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.201 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.042) 0:01:44.098 ******** 2020-10-09T17:11:08.207 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-10-09T17:11:08.209 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.243 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.041) 0:01:44.140 ******** 2020-10-09T17:11:08.250 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:08.252 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.286 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.043) 0:01:44.183 ******** 2020-10-09T17:11:08.314 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-10-09T17:11:08.316 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.348 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.061) 0:01:44.245 ******** 2020-10-09T17:11:08.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-10-09T17:11:08.377 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.409 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.061) 0:01:44.307 ******** 2020-10-09T17:11:08.416 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-10-09T17:11:08.417 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-10-09T17:11:08.454 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.044) 0:01:44.352 ******** 2020-10-09T17:11:08.461 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-10-09T17:11:08.463 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.499 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.044) 0:01:44.396 ******** 2020-10-09T17:11:08.528 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-09T17:11:08.530 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.581 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.081) 0:01:44.478 ******** 2020-10-09T17:11:08.612 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-09T17:11:08.613 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.647 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.066) 0:01:44.544 ******** 2020-10-09T17:11:08.675 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:08.677 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.709 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.062) 0:01:44.607 ******** 2020-10-09T17:11:08.808 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 09 October 2020 17:11:08 +0000 (0:00:00.066) 0:01:44.673 ******** TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-10-09T17:11:08.819 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.043) 0:01:44.716 ******** 2020-10-09T17:11:08.827 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-10-09T17:11:08.829 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.862 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.043) 0:01:44.759 ******** 2020-10-09T17:11:08.869 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-09T17:11:08.870 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.905 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.042) 0:01:44.802 ******** 2020-10-09T17:11:08.912 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-09T17:11:08.914 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:08.948 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:08 +0000 (0:00:00.043) 0:01:44.845 ******** 2020-10-09T17:11:08.976 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-10-09T17:11:08.977 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.010 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.061) 0:01:44.907 ******** 2020-10-09T17:11:09.037 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-09T17:11:09.038 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.071 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.061) 0:01:44.968 ******** 2020-10-09T17:11:09.099 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:09.100 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.134 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.062) 0:01:45.031 ******** 2020-10-09T17:11:09.161 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-09T17:11:09.163 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.202 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.067) 0:01:45.099 ******** 2020-10-09T17:11:09.219 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-10-09T17:11:09.220 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-10-09T17:11:09.255 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.053) 0:01:45.152 ******** 2020-10-09T17:11:09.262 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-10-09T17:11:09.264 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.299 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.044) 0:01:45.197 ******** 2020-10-09T17:11:09.327 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-10-09T17:11:09.329 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.360 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.061) 0:01:45.258 ******** 2020-10-09T17:11:09.387 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-10-09T17:11:09.389 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.421 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.060) 0:01:45.318 ******** 2020-10-09T17:11:09.449 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-10-09T17:11:09.450 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:09.482 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.060) 0:01:45.379 ******** 2020-10-09T17:11:09.575 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-10-09T17:11:09.611 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.128) 0:01:45.508 ******** 2020-10-09T17:11:09.630 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-10-09T17:11:09.631 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-10-09T17:11:09.665 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.053) 0:01:45.562 ******** 2020-10-09T17:11:09.934 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-10-09T17:11:09.935 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-10-09T17:11:09.969 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:09 +0000 (0:00:00.303) 0:01:45.866 ******** 2020-10-09T17:11:09.999 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-10-09T17:11:10.037 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.068) 0:01:45.934 ******** 2020-10-09T17:11:10.064 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-10-09T17:11:10.066 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.105 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.067) 0:01:46.002 ******** 2020-10-09T17:11:10.111 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-10-09T17:11:10.113 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:11:10.152 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.046) 0:01:46.049 ******** 2020-10-09T17:11:10.180 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-10-09T17:11:10.182 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.230 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.078) 0:01:46.127 ******** 2020-10-09T17:11:10.258 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-10-09T17:11:10.260 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.292 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.062) 0:01:46.190 ******** 2020-10-09T17:11:10.320 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-10-09T17:11:10.321 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.354 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.061) 0:01:46.251 ******** 2020-10-09T17:11:10.382 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-10-09T17:11:10.384 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.417 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.062) 0:01:46.314 ******** 2020-10-09T17:11:10.425 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-10-09T17:11:10.426 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-10-09T17:11:10.461 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.043) 0:01:46.358 ******** 2020-10-09T17:11:10.467 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-10-09T17:11:10.469 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.504 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.043) 0:01:46.402 ******** 2020-10-09T17:11:10.533 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:10.535 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-10-09T17:11:10.567 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.062) 0:01:46.465 ******** 2020-10-09T17:11:10.615 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-10-09T17:11:10.616 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.617 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.628 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.629 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.640 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.641 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.652 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-10-09T17:11:10.662 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.673 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.684 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.685 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.696 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.706 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.708 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:10.747 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.179) 0:01:46.644 ******** 2020-10-09T17:11:10.785 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-10-09T17:11:10.786 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-10-09T17:11:10.822 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.075) 0:01:46.719 ******** 2020-10-09T17:11:10.935 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003603", "end": "2020-10-09 17:11:10.911418", "rc": 0, "start": "2020-10-09 17:11:10.907815", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-10-09T17:11:10.993 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:10 +0000 (0:00:00.171) 0:01:46.891 ******** 2020-10-09T17:11:11.021 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.076 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.082) 0:01:46.973 ******** 2020-10-09T17:11:11.106 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.145 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.069) 0:01:47.042 ******** 2020-10-09T17:11:11.153 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-10-09T17:11:11.155 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-10-09T17:11:11.190 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.045) 0:01:47.088 ******** 2020-10-09T17:11:11.196 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-10-09T17:11:11.198 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.234 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.043) 0:01:47.131 ******** 2020-10-09T17:11:11.314 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-10-09T17:11:11.316 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:11:11.350 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.116) 0:01:47.248 ******** 2020-10-09T17:11:11.377 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-10-09T17:11:11.379 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.412 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.061) 0:01:47.309 ******** 2020-10-09T17:11:11.438 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-10-09T17:11:11.440 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.473 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.060) 0:01:47.370 ******** 2020-10-09T17:11:11.499 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-10-09T17:11:11.501 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.533 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.060) 0:01:47.430 ******** 2020-10-09T17:11:11.560 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-10-09T17:11:11.562 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.597 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.063) 0:01:47.494 ******** 2020-10-09T17:11:11.624 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-10-09T17:11:11.626 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.663 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.066) 0:01:47.560 ******** 2020-10-09T17:11:11.701 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-10-09T17:11:11.735 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.072) 0:01:47.633 ******** 2020-10-09T17:11:11.784 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi184.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.796 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.798 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.808 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.810 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.821 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.833 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-10-09T17:11:11.835 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.846 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.848 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.858 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.869 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.880 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.891 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.892 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:11.932 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:11 +0000 (0:00:00.196) 0:01:47.829 ******** 2020-10-09T17:11:12.147 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-10-09T17:11:12.149 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:11:12.185 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.252) 0:01:48.082 ******** 2020-10-09T17:11:12.192 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-10-09T17:11:12.193 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003422", "end": "2020-10-09 17:11:12.130715", "rc": 0, "start": "2020-10-09 17:11:12.127293", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-10-09T17:11:12.235 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.050) 0:01:48.132 ******** 2020-10-09T17:11:12.242 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:12.277 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.041) 0:01:48.174 ******** 2020-10-09T17:11:12.285 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-10-09T17:11:12.286 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:12.329 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.051) 0:01:48.226 ******** 2020-10-09T17:11:12.336 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-09T17:11:12.337 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13639664, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455444, 'size_available': 927561498624, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113889, 'block_size': 4096, 'inode_available': 60941087}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455444, "block_size": 4096, "block_total": 240095108, "block_used": 13639664, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941087, "inode_total": 61054976, "inode_used": 113889, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561498624, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}} 2020-10-09T17:11:12.376 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.047) 0:01:48.273 ******** 2020-10-09T17:11:12.385 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-10-09T17:11:12.387 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:12.422 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.045) 0:01:48.319 ******** 2020-10-09T17:11:12.450 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-10-09T17:11:12.452 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:12.485 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.062) 0:01:48.382 ******** 2020-10-09T17:11:12.514 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-10-09T17:11:12.516 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009809", "end": "2020-10-09 17:11:12.496708", "rc": 0, "start": "2020-10-09 17:11:12.486899", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:12.549 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.064) 0:01:48.446 ******** 2020-10-09T17:11:12.599 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-10-09T17:11:12.601 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13639664, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455444, 'size_available': 927561498624, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113889, 'block_size': 4096, 'inode_available': 60941087}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455444, "block_size": 4096, "block_total": 240095108, "block_used": 13639664, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941087, "inode_total": 61054976, "inode_used": 113889, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561498624, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:12.638 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.088) 0:01:48.535 ******** 2020-10-09T17:11:12.670 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-10-09T17:11:12.714 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:12 +0000 (0:00:00.075) 0:01:48.611 ******** 2020-10-09T17:11:12.763 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-09T17:11:12.766 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:13.393 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-10-09T17:11:13.395 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:11:13.431 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.716) 0:01:49.328 ******** 2020-10-09T17:11:13.458 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-10-09T17:11:13.459 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:13.492 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.061) 0:01:49.389 ******** 2020-10-09T17:11:13.539 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-09T17:11:13.541 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13639655, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455453, 'size_available': 927561535488, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113890, 'block_size': 4096, 'inode_available': 60941086}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455453, "block_size": 4096, "block_total": 240095108, "block_used": 13639655, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941086, "inode_total": 61054976, "inode_used": 113890, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561535488, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}} 2020-10-09T17:11:13.580 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.088) 0:01:49.477 ******** 2020-10-09T17:11:13.712 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-10-09T17:11:13.713 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009799", "end": "2020-10-09 17:11:13.892469", "rc": 0, "start": "2020-10-09 17:11:13.882670", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:13.747 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.167) 0:01:49.645 ******** 2020-10-09T17:11:13.786 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13639655, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455453, 'size_available': 927561535488, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113890, 'block_size': 4096, 'inode_available': 60941086}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455453, "block_size": 4096, "block_total": 240095108, "block_used": 13639655, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941086, "inode_total": 61054976, "inode_used": 113890, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561535488, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:13.821 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.073) 0:01:49.718 ******** 2020-10-09T17:11:13.848 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-10-09T17:11:13.883 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:13 +0000 (0:00:00.062) 0:01:49.780 ******** 2020-10-09T17:11:13.931 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-09T17:11:13.933 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:14.717 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdgySGEolhwAczmC9CO5EcxLlVg5KwHKQT'], 'uuids': ['fcc84c57-b587-4696-bba5-a8ff90e1f590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.825861", "end": "2020-10-09 17:11:14.689886", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdgySGEolhwAczmC9CO5EcxLlVg5KwHKQT"], "labels": [], "masters": [], "uuids": ["fcc84c57-b587-4696-bba5-a8ff90e1f590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:12.864025", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:14.729 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:14.730 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:15.671 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pifXEjxznDKyOWgj60G5YEwkxbjepeftr'], 'uuids': ['2809f8b1-2c32-4459-a683-672c390a8f9c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.622578", "end": "2020-10-09 17:11:15.838890", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pifXEjxznDKyOWgj60G5YEwkxbjepeftr"], "labels": [], "masters": [], "uuids": ["2809f8b1-2c32-4459-a683-672c390a8f9c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:14.216312", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:15.683 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:15.694 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:16.063 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDkepxzYxPx2E4Kx4ruRkKaZ2MlQedl09'], 'uuids': ['28aac1a9-9792-4c6d-9823-ffeb589aabb4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.201335", "end": "2020-10-09 17:11:16.035532", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDkepxzYxPx2E4Kx4ruRkKaZ2MlQedl09"], "labels": [], "masters": [], "uuids": ["28aac1a9-9792-4c6d-9823-ffeb589aabb4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:14.834197", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:16.076 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['2b04c511-aa55-4d14-99bd-e89e73e1b5bb']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["2b04c511-aa55-4d14-99bd-e89e73e1b5bb"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-10-09T17:11:16.078 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-09T17:11:16.955 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83p9WZHOOIk18MXLHbinOU3U24sedTy2Ox8'], 'uuids': ['bfa3adfd-d46e-4c8b-99b2-3ffd8f60638e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.132698", "end": "2020-10-09 17:11:17.125095", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83p9WZHOOIk18MXLHbinOU3U24sedTy2Ox8"], "labels": [], "masters": [], "uuids": ["bfa3adfd-d46e-4c8b-99b2-3ffd8f60638e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:15.992397", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:16.967 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY', 'wwn-0x5000c5009294076e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1', 'wwn-0x5000c5009294076e-part1'], 'uuids': ['2b04c511-aa55-4d14-99bd-e89e73e1b5bb']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294076e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLJY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY", "wwn-0x5000c5009294076e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1", "wwn-0x5000c5009294076e-part1"], "labels": [], "masters": [], "uuids": ["2b04c511-aa55-4d14-99bd-e89e73e1b5bb"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLJY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294076e"}}, "skip_reason": 2020-10-09T17:11:16.969 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-09T17:11:17.353 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-SwYoLE-KX6Q-2rgN-1XIW-SWuo-TNd8-k8znMk', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.151787", "end": "2020-10-09 17:11:17.322121", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-SwYoLE-KX6Q-2rgN-1XIW-SWuo-TNd8-k8znMk", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:16.170334", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-10-09T17:11:17.355 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:17.365 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:17.366 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:18.297 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-U41TO4-jo05-qvjx-I67r-ckzY-XGjS-gKIth8', 'nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN', 'nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.200239", "end": "2020-10-09 17:11:18.465196", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-U41TO4-jo05-qvjx-I67r-ckzY-XGjS-gKIth8", "nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN", "nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:17.264957", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-10-09T17:11:18.299 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:18.309 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:18.320 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:18.625 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdnAE1frFZkU8ahkv7FGTvl2nQyXnfRr94'], 'uuids': ['268073a1-99c4-4271-8135-f3827650d35c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.147555", "end": "2020-10-09 17:11:18.602751", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdnAE1frFZkU8ahkv7FGTvl2nQyXnfRr94"], "labels": [], "masters": [], "uuids": ["268073a1-99c4-4271-8135-f3827650d35c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:17.455196", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:18.636 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:18.648 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:19.590 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pe5220uYPC0XolHTqttd70WXrUL3YahIh'], 'uuids': ['c600e107-1d0c-4068-96a7-6779131ef113']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.143067", "end": "2020-10-09 17:11:19.751581", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pe5220uYPC0XolHTqttd70WXrUL3YahIh"], "labels": [], "masters": [], "uuids": ["c600e107-1d0c-4068-96a7-6779131ef113"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:18.608514", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:19.592 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:19.602 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:19.886 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0Gehdq6W6ardaYE4omkNSwyr5v8jeyRhOKI0u'], 'uuids': ['c9077083-d10b-49e6-b5bf-fd66fd38aac3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.131572", "end": "2020-10-09 17:11:19.868748", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0Gehdq6W6ardaYE4omkNSwyr5v8jeyRhOKI0u"], "labels": [], "masters": [], "uuids": ["c9077083-d10b-49e6-b5bf-fd66fd38aac3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:18.737176", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:19.898 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:20.833 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pcLjhUa7d2244T0d1c4Av8JAaNyTe8SzR'], 'uuids': ['ecd104da-5999-4e81-93fa-e1cd1ba865e6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.113805", "end": "2020-10-09 17:11:21.009635", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pcLjhUa7d2244T0d1c4Av8JAaNyTe8SzR"], "labels": [], "masters": [], "uuids": ["ecd104da-5999-4e81-93fa-e1cd1ba865e6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:19.895830", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:20.845 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:21.148 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDr9w87ZJptgW036U7K9RCKnxTCowcWw1'], 'uuids': ['d22ac400-399c-4076-9536-577f92baec55']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.133570", "end": "2020-10-09 17:11:21.127902", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDr9w87ZJptgW036U7K9RCKnxTCowcWw1"], "labels": [], "masters": [], "uuids": ["d22ac400-399c-4076-9536-577f92baec55"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:19.994332", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:21.195 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:21 +0000 (0:00:07.312) 0:01:57.092 ******** 2020-10-09T17:11:21.245 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:21.247 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:21.484 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdgySGEolhwAczmC9CO5EcxLlVg5KwHKQT'], 'uuids': ['fcc84c57-b587-4696-bba5-a8ff90e1f590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.112218", "end": "2020-10-09 17:11:21.455409", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdgySGEolhwAczmC9CO5EcxLlVg5KwHKQT"], "labels": [], "masters": [], "uuids": ["fcc84c57-b587-4696-bba5-a8ff90e1f590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:21.343191", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.107412 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.107412 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:21.496 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:21.497 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:21.727 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDkepxzYxPx2E4Kx4ruRkKaZ2MlQedl09'], 'uuids': ['28aac1a9-9792-4c6d-9823-ffeb589aabb4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.107925", "end": "2020-10-09 17:11:21.705338", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDkepxzYxPx2E4Kx4ruRkKaZ2MlQedl09"], "labels": [], "masters": [], "uuids": ["28aac1a9-9792-4c6d-9823-ffeb589aabb4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:21.597413", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.104834 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.104834 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:21.738 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['2b04c511-aa55-4d14-99bd-e89e73e1b5bb']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["2b04c511-aa55-4d14-99bd-e89e73e1b5bb"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-10-09T17:11:21.740 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-09T17:11:21.917 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-SwYoLE-KX6Q-2rgN-1XIW-SWuo-TNd8-k8znMk', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053255", "end": "2020-10-09 17:11:21.886874", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-SwYoLE-KX6Q-2rgN-1XIW-SWuo-TNd8-k8znMk", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:21.833619", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0488897 s, 2.4 GB/s", "stderr_lines": [" 2020-10-09T17:11:21.919 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0488897 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:21.928 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:21.940 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.106 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-10-09T17:11:22.108 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pBdqIMSP71exL15RsfHKJ6T9yVuOz4kC8'], 'uuids': ['fea14744-f29f-42ec-beda-0e7139f12ab2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.140633", "end": "2020-10-09 17:11:22.276388", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pBdqIMSP71exL15RsfHKJ6T9yVuOz4kC8"], "labels": [], "masters": [], "uuids": ["fea14744-f29f-42ec-beda-0e7139f12ab2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:21.135755", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-10-09T17:11:22.151 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:22 +0000 (0:00:00.955) 0:01:58.048 ******** 2020-10-09T17:11:22.168 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:22.170 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdnAE1frFZkU8ahkv7FGTvl2nQyXnfRr94'], 'uuids': ['268073a1-99c4-4271-8135-f3827650d35c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.108662", "end": "2020-10-09 17:11:22.139850", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdnAE1frFZkU8ahkv7FGTvl2nQyXnfRr94"], "labels": [], "masters": [], "uuids": ["268073a1-99c4-4271-8135-f3827650d35c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:22.031188", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.104495 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.104495 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': ' 2020-10-09T17:11:22.172 INFO:teuthology.task.ansible.out:0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.183 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.205 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.414 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0Gehdq6W6ardaYE4omkNSwyr5v8jeyRhOKI0u'], 'uuids': ['c9077083-d10b-49e6-b5bf-fd66fd38aac3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.106923", "end": "2020-10-09 17:11:22.383731", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0Gehdq6W6ardaYE4omkNSwyr5v8jeyRhOKI0u"], "labels": [], "masters": [], "uuids": ["c9077083-d10b-49e6-b5bf-fd66fd38aac3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:22.276808", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102235 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102235 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:22.426 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.500 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pifXEjxznDKyOWgj60G5YEwkxbjepeftr'], 'uuids': ['2809f8b1-2c32-4459-a683-672c390a8f9c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.160618", "end": "2020-10-09 17:11:22.664925", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pifXEjxznDKyOWgj60G5YEwkxbjepeftr"], "labels": [], "masters": [], "uuids": ["2809f8b1-2c32-4459-a683-672c390a8f9c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:22.504307", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155804 s, 740 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155804 s, 740 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:22.501 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.513 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:22.650 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDr9w87ZJptgW036U7K9RCKnxTCowcWw1'], 'uuids': ['d22ac400-399c-4076-9536-577f92baec55']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.111307", "end": "2020-10-09 17:11:22.629445", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-b236f5EvwrbFdTFDq2m301UBGWd0GehdDr9w87ZJptgW036U7K9RCKnxTCowcWw1"], "labels": [], "masters": [], "uuids": ["d22ac400-399c-4076-9536-577f92baec55"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:22.518138", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106588 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106588 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:22.705 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:22 +0000 (0:00:00.554) 0:01:58.602 ******** 2020-10-09T17:11:22.786 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83p9WZHOOIk18MXLHbinOU3U24sedTy2Ox8'], 'uuids': ['bfa3adfd-d46e-4c8b-99b2-3ffd8f60638e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.158899", "end": "2020-10-09 17:11:22.961221", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83p9WZHOOIk18MXLHbinOU3U24sedTy2Ox8"], "labels": [], "masters": [], "uuids": ["bfa3adfd-d46e-4c8b-99b2-3ffd8f60638e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:22.802322", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154193 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154193 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:22.798 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY', 'wwn-0x5000c5009294076e'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1', 'wwn-0x5000c5009294076e-part1'], 'uuids': ['2b04c511-aa55-4d14-99bd-e89e73e1b5bb']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c5009294076e', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XLJY', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY", "wwn-0x5000c5009294076e"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLJY-part1", "wwn-0x5000c5009294076e-part1"], "labels": [], "masters": [], "uuids": ["2b04c511-aa55-4d14-99bd-e89e73e1b5bb"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLJY", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294076e"}}, "skip_reason": 2020-10-09T17:11:22.800 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-10-09T17:11:22.977 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-U41TO4-jo05-qvjx-I67r-ckzY-XGjS-gKIth8', 'nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN', 'nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.050605", "end": "2020-10-09 17:11:23.142355", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-U41TO4-jo05-qvjx-I67r-ckzY-XGjS-gKIth8", "nvme-INTEL_SSDPEDMD400G4_CVFT626500CE400BGN", "nvme-nvme.8086-43564654363236353030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:23.091750", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0457815 s, 2.5 GB/s", "std 2020-10-09T17:11:22.978 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0457815 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.135702", "end": "2020-10-09 17:11:22.950037", "rc": 0, "start": "2020-10-09 17:11:22.814335", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:23.014 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.309) 0:01:58.912 ******** 2020-10-09T17:11:23.021 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:23.023 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.191 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-10-09T17:11:23.193 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042427", "end": "2020-10-09 17:11:23.168464", "rc": 0, "start": "2020-10-09 17:11:23.126037", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:23.225 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.210) 0:01:59.122 ******** 2020-10-09T17:11:23.254 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-10-09T17:11:23.291 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.065) 0:01:59.188 ******** 2020-10-09T17:11:23.299 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:23.302 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pe5220uYPC0XolHTqttd70WXrUL3YahIh'], 'uuids': ['c600e107-1d0c-4068-96a7-6779131ef113']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.154242", "end": "2020-10-09 17:11:23.433806", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pe5220uYPC0XolHTqttd70WXrUL3YahIh"], "labels": [], "masters": [], "uuids": ["c600e107-1d0c-4068-96a7-6779131ef113"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:23.279564", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149995 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149995 s, 769 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': ' 2020-10-09T17:11:23.303 INFO:teuthology.task.ansible.out:0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.313 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-10-09T17:11:23.315 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.349 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.058) 0:01:59.246 ******** 2020-10-09T17:11:23.376 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-10-09T17:11:23.378 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.409 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.060) 0:01:59.307 ******** 2020-10-09T17:11:23.437 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-09T17:11:23.438 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.475 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.065) 0:01:59.372 ******** 2020-10-09T17:11:23.502 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-10-09T17:11:23.503 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.535 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.060) 0:01:59.433 ******** 2020-10-09T17:11:23.564 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:23.566 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pcLjhUa7d2244T0d1c4Av8JAaNyTe8SzR'], 'uuids': ['ecd104da-5999-4e81-93fa-e1cd1ba865e6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.163078", "end": "2020-10-09 17:11:23.736373", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pcLjhUa7d2244T0d1c4Av8JAaNyTe8SzR"], "labels": [], "masters": [], "uuids": ["ecd104da-5999-4e81-93fa-e1cd1ba865e6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:23.573295", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158328 s, 729 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158328 s, 729 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:23.576 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.660 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-10-09T17:11:23.662 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:11:23.696 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.160) 0:01:59.593 ******** 2020-10-09T17:11:23.734 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-09T17:11:23.736 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13639664, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455444, 'size_available': 927561498624, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113889, 'block_size': 4096, 'inode_available': 60941087}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455444, "block_size": 4096, "block_total": 240095108, "block_used": 13639664, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941087, "inode_total": 61054976, "inode_used": 113889, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561498624, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.771 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.074) 0:01:59.668 ******** 2020-10-09T17:11:23.798 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-10-09T17:11:23.800 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.837 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.065) 0:01:59.734 ******** 2020-10-09T17:11:23.859 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-10-09T17:11:23.861 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pBdqIMSP71exL15RsfHKJ6T9yVuOz4kC8'], 'uuids': ['fea14744-f29f-42ec-beda-0e7139f12ab2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.153423", "end": "2020-10-09 17:11:24.021463", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-zwivOhbH6t0M6H1FnWiDSRCDEpUfF83pBdqIMSP71exL15RsfHKJ6T9yVuOz4kC8"], "labels": [], "masters": [], "uuids": ["fea14744-f29f-42ec-beda-0e7139f12ab2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-10-09 17:11:23.868040", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148747 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148747 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:23.870 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:23.871 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.903 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.066) 0:01:59.800 ******** 2020-10-09T17:11:23.936 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.032) 0:01:59.833 ******** 2020-10-09T17:11:23.942 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:23.944 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:23.978 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:23 +0000 (0:00:00.042) 0:01:59.875 ******** 2020-10-09T17:11:24.007 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.040 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.061) 0:01:59.937 ******** 2020-10-09T17:11:24.067 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-10-09T17:11:24.069 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.100 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.060) 0:01:59.997 ******** 2020-10-09T17:11:24.128 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-10-09T17:11:24.130 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.162 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.061) 0:02:00.059 ******** 2020-10-09T17:11:24.213 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-10-09T17:11:24.215 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.139740", "end": "2020-10-09 17:11:24.376834", "rc": 0, "start": "2020-10-09 17:11:24.237094", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:24.247 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.085) 0:02:00.144 ******** 2020-10-09T17:11:24.441 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-10-09T17:11:24.443 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.060088", "end": "2020-10-09 17:11:24.611249", "rc": 0, "start": "2020-10-09 17:11:24.551161", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:24.479 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.232) 0:02:00.376 ******** 2020-10-09T17:11:24.507 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-10-09T17:11:24.545 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.065) 0:02:00.442 ******** 2020-10-09T17:11:24.572 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-10-09T17:11:24.574 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.608 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.062) 0:02:00.505 ******** 2020-10-09T17:11:24.634 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-10-09T17:11:24.636 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.668 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.059) 0:02:00.565 ******** 2020-10-09T17:11:24.695 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.730 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.062) 0:02:00.627 ******** 2020-10-09T17:11:24.757 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-10-09T17:11:24.759 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-10-09T17:11:24.795 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.065) 0:02:00.692 ******** 2020-10-09T17:11:24.802 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-10-09T17:11:24.804 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:24.838 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.042) 0:02:00.735 ******** 2020-10-09T17:11:24.959 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-10-09T17:11:24.961 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-10-09T17:11:24.995 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:24 +0000 (0:00:00.157) 0:02:00.892 ******** 2020-10-09T17:11:25.034 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-10-09T17:11:25.036 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'block_used': 13639655, 'uuid': '2b04c511-aa55-4d14-99bd-e89e73e1b5bb', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226455453, 'size_available': 927561535488, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113890, 'block_size': 4096, 'inode_available': 60941086}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226455453, "block_size": 4096, "block_total": 240095108, "block_used": 13639655, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941086, "inode_total": 61054976, "inode_used": 113890, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927561535488, "size_total": 983429562368, "uuid": "2b04c511-aa55-4d14-99bd-e89e73e1b5bb"}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.070 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.074) 0:02:00.967 ******** 2020-10-09T17:11:25.097 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-10-09T17:11:25.098 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.130 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.060) 0:02:01.028 ******** 2020-10-09T17:11:25.158 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.192 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.061) 0:02:01.089 ******** 2020-10-09T17:11:25.219 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.253 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.060) 0:02:01.150 ******** 2020-10-09T17:11:25.281 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-10-09T17:11:25.282 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.314 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.061) 0:02:01.211 ******** 2020-10-09T17:11:25.341 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.343 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:25.376 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.061) 0:02:01.273 ******** 2020-10-09T17:11:25.404 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-10-09T17:11:25.406 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:25.439 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.062) 0:02:01.336 ******** 2020-10-09T17:11:25.575 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-09T17:11:25.577 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:25.776 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-10-09T17:11:25.778 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-10-09T17:11:25.813 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:25 +0000 (0:00:00.374) 0:02:01.710 ******** 2020-10-09T17:11:25.820 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-09T17:11:25.822 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.033 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.121 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.278 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.314 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:26 +0000 (0:00:00.501) 0:02:02.212 ******** 2020-10-09T17:11:26.353 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.460 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-10-09T17:11:26.494 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:26 +0000 (0:00:00.180) 0:02:02.392 ******** 2020-10-09T17:11:26.586 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-09T17:11:26.593 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.651 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-09T17:11:26.652 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:26.769 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:26.854 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-10-09T17:11:26.856 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:26.886 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-09T17:11:26.888 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:26.995 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:27.006 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:27.042 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.547) 0:02:02.940 ******** 2020-10-09T17:11:27.103 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-10-09T17:11:27.146 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.104) 0:02:03.044 ******** 2020-10-09T17:11:27.176 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-10-09T17:11:27.178 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:11:27.216 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.068) 0:02:03.113 ******** 2020-10-09T17:11:27.248 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-10-09T17:11:27.250 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-10-09T17:11:27.291 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.075) 0:02:03.188 ******** 2020-10-09T17:11:27.298 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-10-09T17:11:27.300 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-10-09T17:11:27.344 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.052) 0:02:03.241 ******** 2020-10-09T17:11:27.500 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-09T17:11:27.502 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:27.618 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:27.714 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-10-09T17:11:27.716 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true} 2020-10-09T17:11:27.754 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.410) 0:02:03.651 ******** 2020-10-09T17:11:27.761 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-10-09T17:11:27.763 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:27.847 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-10-09T17:11:27.861 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:27.903 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:27 +0000 (0:00:00.148) 0:02:03.800 ******** 2020-10-09T17:11:28.046 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-10-09T17:11:28.048 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:11:28.088 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:28 +0000 (0:00:00.185) 0:02:03.985 ******** 2020-10-09T17:11:28.126 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-10-09T17:11:28.128 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-10-09T17:11:28.162 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:28 +0000 (0:00:00.073) 0:02:04.059 ******** 2020-10-09T17:11:29.195 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-10-09T17:11:29.197 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-10-09T17:11:29.232 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:01.069) 0:02:05.129 ******** 2020-10-09T17:11:29.386 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-10-09T17:11:29.388 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi184.front.sepia.ceph.com] => {"changed": true} 2020-10-09T17:11:29.424 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.191) 0:02:05.321 ******** 2020-10-09T17:11:29.457 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.033) 0:02:05.354 ******** 2020-10-09T17:11:29.559 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-10-09T17:11:29.561 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:11:29.597 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.140) 0:02:05.495 ******** 2020-10-09T17:11:29.659 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-10-09T17:11:29.661 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-10-09T17:11:29.697 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.099) 0:02:05.595 ******** 2020-10-09T17:11:29.833 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-10-09T17:11:29.866 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.168) 0:02:05.764 ******** 2020-10-09T17:11:29.930 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-10-09T17:11:29.932 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-10-09T17:11:29.968 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:29 +0000 (0:00:00.101) 0:02:05.865 ******** 2020-10-09T17:11:30.032 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-10-09T17:11:30.044 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:30 +0000 (0:00:00.075) 0:02:05.941 ******** 2020-10-09T17:11:30.300 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-10-09T17:11:30.302 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-10-09T17:11:30.340 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:30 +0000 (0:00:00.295) 0:02:06.237 ******** 2020-10-09T17:11:30.379 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-10-09T17:11:30.380 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1230, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-10-09T17:11:30.417 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:30 +0000 (0:00:00.077) 0:02:06.314 ******** 2020-10-09T17:11:30.741 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-10-09T17:11:30.765 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-10-09T17:11:30.775 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:30 +0000 (0:00:00.357) 0:02:06.672 ******** 2020-10-09T17:11:31.271 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-10-09T17:11:31.273 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:11:31.306 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.530) 0:02:07.203 ******** 2020-10-09T17:11:31.479 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-10-09T17:11:31.481 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047854", "end": "2020-10-09 17:11:31.462124", "rc": 0, "start": "2020-10-09 17:11:31.414270", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-10-09T17:11:31.514 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.208) 0:02:07.412 ******** 2020-10-09T17:11:31.542 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-10-09T17:11:31.544 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:31.585 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.070) 0:02:07.482 ******** 2020-10-09T17:11:31.624 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:31.664 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.078) 0:02:07.561 ******** 2020-10-09T17:11:31.673 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-10-09T17:11:31.675 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:11:31.709 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.045) 0:02:07.606 ******** 2020-10-09T17:11:31.884 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045437", "end": "2020-10-09 17:11:32.060742", "rc": 0, "start": "2020-10-09 17:11:32.015305", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-10-09T17:11:31.921 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.211) 0:02:07.818 ******** 2020-10-09T17:11:31.959 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-10-09T17:11:31.961 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:31.994 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:31 +0000 (0:00:00.072) 0:02:07.891 ******** 2020-10-09T17:11:32.021 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-10-09T17:11:32.023 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:32.055 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:32 +0000 (0:00:00.061) 0:02:07.952 ******** 2020-10-09T17:11:32.531 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-10-09T17:11:32.533 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:11:32.567 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:32 +0000 (0:00:00.512) 0:02:08.464 ******** 2020-10-09T17:11:32.595 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:32.631 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:32 +0000 (0:00:00.064) 0:02:08.529 ******** 2020-10-09T17:11:32.935 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:11:32.970 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:32 +0000 (0:00:00.338) 0:02:08.867 ******** 2020-10-09T17:11:32.999 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-10-09T17:11:33.001 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.033 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.062) 0:02:08.930 ******** 2020-10-09T17:11:33.040 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-10-09T17:11:33.042 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-10-09T17:11:33.081 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.048) 0:02:08.979 ******** 2020-10-09T17:11:33.120 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-10-09T17:11:33.122 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.159 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.078) 0:02:09.057 ******** 2020-10-09T17:11:33.393 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-10-09T17:11:33.395 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:19 UTC", "ActiveEnterTimestampMonotonic": "24438822", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket system.slice sysinit.target systemd-tmpfiles-setup.service network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:13 UTC", "AssertTimestampMonotonic": "17888131", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:13 UTC", "ConditionTimestampMonotonic": "17888130", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-10-09T17:11:33.397 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2194", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:19 UTC", "ExecMainStartTimestampMonotonic": "24438797", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:13 UTC", "InactiveExitTimestampMonotonic": "17889030", "InvocationID": "ec7cdbb8879246cab2c4d15d7eed6fd3", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-10-09T17:11:33.398 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2194", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:19 UTC", "S 2020-10-09T17:11:33.400 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24438822", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:19 UTC", "WatchdogTimestampMonotonic": "24438820", "WatchdogUSec": "0"}} 2020-10-09T17:11:33.401 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-10-09T17:11:33.403 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-10-09T17:11:33.439 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.279) 0:02:09.336 ******** 2020-10-09T17:11:33.474 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.035) 0:02:09.371 ******** 2020-10-09T17:11:33.489 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-10-09T17:11:33.521 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.047) 0:02:09.418 ******** 2020-10-09T17:11:33.529 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-10-09T17:11:33.532 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.572 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.050) 0:02:09.469 ******** 2020-10-09T17:11:33.578 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:33.582 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.615 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.043) 0:02:09.512 ******** 2020-10-09T17:11:33.642 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-10-09T17:11:33.644 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.676 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.061) 0:02:09.573 ******** 2020-10-09T17:11:33.795 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-10-09T17:11:33.796 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ActiveEnterTimestampMonotonic": "17589812", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket -.mount systemd-tmpfiles-setup.service sysinit.target network.target basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2020-10-09 17:08:36 UTC", "AssertTimestampMonotonic": "17408128", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ConditionTimestampMonotonic": "17408127", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-10-09T17:11:33.798 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1010", "ExecMainStartTimestamp": "Fri 2020-10-09 17:08:36 UTC", "ExecMainStartTimestampMonotonic": "17589803", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2020-10-09 17:08:36 UTC", "InactiveExitTimestampMonotonic": "17408972", "InvocationID": "c38228f1b27047689ece162f1eae4bf7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-10-09T17:11:33.801 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1010", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2020-10-09 17:08:36 UTC", "S 2020-10-09T17:11:33.804 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "17589812", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2020-10-09 17:08:36 UTC", "WatchdogTimestampMonotonic": "17589812", "WatchdogUSec": "0"}} 2020-10-09T17:11:33.834 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.157) 0:02:09.731 ******** 2020-10-09T17:11:33.862 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:33.864 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-10-09T17:11:33.897 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.063) 0:02:09.794 ******** 2020-10-09T17:11:33.924 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:33.926 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:33.959 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:33 +0000 (0:00:00.062) 0:02:09.857 ******** 2020-10-09T17:11:33.987 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-10-09T17:11:33.989 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:34.044 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:34 +0000 (0:00:00.084) 0:02:09.942 ******** 2020-10-09T17:11:34.555 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-10-09T17:11:34.556 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1602263420, "cache_updated": false, "changed": false} 2020-10-09T17:11:34.598 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:34 +0000 (0:00:00.552) 0:02:10.494 ******** 2020-10-09T17:11:34.763 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-10-09T17:11:34.764 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:34.916 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:34 +0000 (0:00:00.319) 0:02:10.813 ******** 2020-10-09T17:11:34.922 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-10-09T17:11:34.924 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"cache_update_time": 1602263444, "cache_updated": false, "changed": false} 2020-10-09T17:11:34.959 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:34 +0000 (0:00:00.042) 0:02:10.856 ******** 2020-10-09T17:11:34.987 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-10-09T17:11:34.988 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:35.020 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:35 +0000 (0:00:00.061) 0:02:10.917 ******** 2020-10-09T17:11:35.038 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-10-09T17:11:35.039 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-10-09T17:11:35.075 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:35 +0000 (0:00:00.054) 0:02:10.972 ******** 2020-10-09T17:11:35.145 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-10-09T17:11:35.183 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:35 +0000 (0:00:00.108) 0:02:11.081 ******** 2020-10-09T17:11:35.399 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-10-09T17:11:35.445 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:35 +0000 (0:00:00.261) 0:02:11.343 ******** 2020-10-09T17:11:35.508 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1230, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-10-09T17:11:35.547 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:35 +0000 (0:00:00.101) 0:02:11.444 ******** 2020-10-09T17:11:36.349 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-10-09T17:11:36.351 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-10-09T17:11:36.384 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:36 +0000 (0:00:00.836) 0:02:12.281 ******** 2020-10-09T17:11:36.391 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-10-09T17:11:36.428 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:36 +0000 (0:00:00.043) 0:02:12.325 ******** 2020-10-09T17:11:36.868 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-10-09T17:11:36.869 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false} 2020-10-09T17:11:36.901 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:36 +0000 (0:00:00.473) 0:02:12.799 ******** 2020-10-09T17:11:36.929 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-10-09T17:11:36.930 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-10-09T17:11:36.965 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:36 +0000 (0:00:00.063) 0:02:12.862 ******** 2020-10-09T17:11:36.974 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false} 2020-10-09T17:11:37.009 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.043) 0:02:12.906 ******** 2020-10-09T17:11:37.038 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-10-09T17:11:37.072 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.062) 0:02:12.969 ******** 2020-10-09T17:11:37.100 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-10-09T17:11:37.102 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1602263337.4921489, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1602263337.488149, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10413, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1602263337.488149, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3466829222", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-10-09T17:11:37.136 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.064) 0:02:13.033 ******** 2020-10-09T17:11:37.217 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1602263309.9805348, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1602263308.528535, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10413, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1602263308.528535, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3466829222", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-10-09T17:11:37.255 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.118) 0:02:13.152 ******** 2020-10-09T17:11:37.387 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-10-09T17:11:37.389 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi184.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-10-09T17:11:37.422 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.167) 0:02:13.319 ******** 2020-10-09T17:11:37.454 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.031) 0:02:13.351 ******** 2020-10-09T17:11:37.460 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-10-09T17:11:37.462 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-10-09T17:11:37.495 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.041) 0:02:13.392 ******** 2020-10-09T17:11:37.501 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-10-09T17:11:37.536 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.041) 0:02:13.433 ******** 2020-10-09T17:11:37.542 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-10-09T17:11:37.544 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:37.577 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.041) 0:02:13.474 ******** 2020-10-09T17:11:37.584 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:37.622 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.045) 0:02:13.519 ******** 2020-10-09T17:11:37.712 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-10-09T17:11:37.714 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-10-09T17:11:37.748 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.126) 0:02:13.645 ******** 2020-10-09T17:11:37.755 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-10-09T17:11:37.791 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.043) 0:02:13.689 ******** 2020-10-09T17:11:37.872 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-10-09T17:11:37.874 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-10-09T17:11:37.907 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.115) 0:02:13.805 ******** 2020-10-09T17:11:37.925 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-10-09T17:11:37.959 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.051) 0:02:13.856 ******** 2020-10-09T17:11:37.966 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.000 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:37 +0000 (0:00:00.041) 0:02:13.898 ******** 2020-10-09T17:11:38.007 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.043 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.042) 0:02:13.940 ******** 2020-10-09T17:11:38.050 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-10-09T17:11:38.051 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.053 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.097 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.054) 0:02:13.995 ******** 2020-10-09T17:11:38.105 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.107 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.107 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.144 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.046) 0:02:14.041 ******** 2020-10-09T17:11:38.151 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-10-09T17:11:38.152 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.153 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.153 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result 2020-10-09T17:11:38.155 INFO:teuthology.task.ansible.out: was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.190 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.046) 0:02:14.087 ******** 2020-10-09T17:11:38.198 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.199 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.200 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi184.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.235 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.044) 0:02:14.132 ******** 2020-10-09T17:11:38.241 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-10-09T17:11:38.243 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.279 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.043) 0:02:14.176 ******** 2020-10-09T17:11:38.286 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.333 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.053) 0:02:14.230 ******** 2020-10-09T17:11:38.341 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.378 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.045) 0:02:14.275 ******** 2020-10-09T17:11:38.387 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.423 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.044) 0:02:14.320 ******** 2020-10-09T17:11:38.429 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-10-09T17:11:38.431 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.467 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.044) 0:02:14.364 ******** 2020-10-09T17:11:38.474 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.510 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.043) 0:02:14.408 ******** 2020-10-09T17:11:38.517 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-10-09T17:11:38.519 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.553 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.042) 0:02:14.450 ******** 2020-10-09T17:11:38.561 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.595 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.042) 0:02:14.492 ******** 2020-10-09T17:11:38.601 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-10-09T17:11:38.603 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.639 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.043) 0:02:14.536 ******** 2020-10-09T17:11:38.645 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.683 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.044) 0:02:14.580 ******** 2020-10-09T17:11:38.690 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.727 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.043) 0:02:14.624 ******** 2020-10-09T17:11:38.733 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.769 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.042) 0:02:14.667 ******** 2020-10-09T17:11:38.776 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-10-09T17:11:38.778 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.817 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.047) 0:02:14.714 ******** 2020-10-09T17:11:38.824 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.861 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.044) 0:02:14.758 ******** 2020-10-09T17:11:38.867 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-10-09T17:11:38.869 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.902 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.041) 0:02:14.800 ******** 2020-10-09T17:11:38.910 INFO:teuthology.task.ansible.out:skipping: [smithi184.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-10-09T17:11:38.945 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:38 +0000 (0:00:00.042) 0:02:14.842 ******** 2020-10-09T17:11:39.055 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-10-09T17:11:39.057 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011088", "end": "2020-10-09 17:11:39.027483", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-10-09 17:11:39.016395", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.098 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010444", "end": "2020-10-09 17:11:39.263709", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-10-09 17:11:39.253265", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.173 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006155", "end": "2020-10-09 17:11:39.150771", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-10-09 17:11:39.144616", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.174 INFO:teuthology.task.ansible.out: 2020-10-09T17:11:39.206 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005778", "end": "2020-10-09 17:11:39.381215", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-10-09 17:11:39.375437", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.291 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005372", "end": "2020-10-09 17:11:39.270601", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-10-09 17:11:39.265229", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.322 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005475", "end": "2020-10-09 17:11:39.496079", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-10-09 17:11:39.490604", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.417 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.017745", "end": "2020-10-09 17:11:39.395222", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-10-09 17:11:39.377477", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.454 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:39 +0000 (0:00:00.508) 0:02:15.351 ******** 2020-10-09T17:11:39.462 INFO:teuthology.task.ansible.out:ok: [smithi184.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025617", "end": "2020-10-09 17:11:39.627501", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-10-09 17:11:39.601884", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-10-09T17:11:39.499 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:39 +0000 (0:00:00.045) 0:02:15.396 ******** 2020-10-09T17:11:39.505 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-10-09T17:11:39.541 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:39 +0000 (0:00:00.041) 0:02:15.438 ******** 2020-10-09T17:11:39.584 INFO:teuthology.task.ansible.out:Friday 09 October 2020 17:11:39 +0000 (0:00:00.042) 0:02:15.480 ******** 2020-10-09T17:11:39.663 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-10-09T17:11:39.665 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-10-09T17:11:39.732 INFO:teuthology.task.ansible.out:changed: [smithi184.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-10-09T17:11:39.826 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-10-09T17:11:39.829 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-10-09T17:11:39.833 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2020-10-09T17:11:39.835 INFO:teuthology.task.ansible.out:smithi060.front.sepia.ceph.com : ok=114 changed=30 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi184.front.sepia.ceph.com : ok=114 changed=30 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Friday 09 October 2020 17:11:39 +0000 (0:00:00.249) 0:02:15.730 ******** =============================================================================== users : Create all admin users with sudo access. ----------------------- 17.14s users : Update authorized_keys using the keys repo --------------------- 16.65s testnode : Ensure packages are not present. ----------------------------- 8.29s testnode : Install packages --------------------------------------------- 7.97s testnode : Zap all non-root disks --------------------------------------- 7.31s users : Remove revoked users -------------------------------------------- 5.16s common : Update apt cache ----------------------------------------------- 2.85s testnode : Update apt cache. -------------------------------------------- 2.56s Gathering Facts --------------------------------------------------------- 1.74s testnode : Install apt keys --------------------------------------------- 1.58s users : Update authorized_keys for each user with literal keys ---------- 1.14s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 1.07s testnode : Install packages via pip ------------------------------------- 0.84s users : Clone the keys repo --------------------------------------------- 0.82s Gathering Facts --------------------------------------------------------- 0.80s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.62s testnode : Install python-apt ------------------------------------------- 0.61s ansible-managed : Create the ansible user. ------------------------------ 0.57s testnode : Set the system's hostname ------------------------------------ 0.56s ansible-managed : Add authorized keys for the ans 2020-10-09T17:11:39.836 INFO:teuthology.task.ansible.out:ible user. ------------- 0.56s 2020-10-09T17:11:40.172 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi184.front.sepia.ceph.com')] 2020-10-09T17:11:40.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-09T17:11:40.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:11:40.254 INFO:teuthology.orchestra.run.smithi060:> true 2020-10-09T17:11:40.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-10-09T17:11:40.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi184.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-10-09T17:11:40.523 INFO:teuthology.orchestra.run.smithi184:> true 2020-10-09T17:11:40.711 INFO:teuthology.run_tasks:Running task clock... 2020-10-09T17:11:40.722 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-10-09T17:11:40.722 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-10-09T17:11:40.723 INFO:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-10-09T17:11:40.778 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-10-09T17:11:40.779 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Command line: ntpd -gq 2020-10-09T17:11:40.779 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: proto: precision = 0.126 usec (-23) 2020-10-09T17:11:40.780 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-09T17:11:40.781 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2020-10-09T17:11:40.781 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-09T17:11:40.781 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: restrict ::: KOD does nothing without LIMITED. 2020-10-09T17:11:40.781 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen and drop on 0 v6wildcard [::]:123 2020-10-09T17:11:40.781 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-10-09T17:11:40.782 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen normally on 2 lo 127.0.0.1:123 2020-10-09T17:11:40.782 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen normally on 3 ens1f0 172.21.15.60:123 2020-10-09T17:11:40.783 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen normally on 4 lo [::1]:123 2020-10-09T17:11:40.783 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%3]:123 2020-10-09T17:11:40.783 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:40 ntpd[9110]: Listening on routing socket on fd #22 for interface updates 2020-10-09T17:11:41.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:41.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:41.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:41.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:41.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:41.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:41.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:41.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:41 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:43.779 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:43.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:43.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:43.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:43.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:43.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:43.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:43.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:43 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:44.779 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:44 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:44.779 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:44 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:44.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:44 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:44.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:44 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:45.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:45.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:45.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:45.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:45.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:45.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:45.782 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:45 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:46.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:46 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:46.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:46 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:46.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:46 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:46.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:46 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:47.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:47.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:47.780 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:47.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:47.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-10-09T17:11:47.781 INFO:teuthology.orchestra.run.smithi060.stderr: 9 Oct 17:11:47 ntpd[9110]: can't open /var/log/ntpstats/loopstats.20201009: Permission denied 2020-10-09T17:11:47.781 INFO:teuthology.orchestra.run.smithi060.stdout: 9 Oct 17:11:47 ntpd[9110]: ntpd: time slew -0.001806 s 2020-10-09T17:11:47.782 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew -0.001806s 2020-10-09T17:11:47.833 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2020-10-09T17:11:47.833 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2020-10-09T17:11:47.834 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:47.834 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:47.834 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:47.834 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:47.835 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-10-09T17:11:47.835 INFO:teuthology.orchestra.run.smithi184:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-10-09T17:11:47.894 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-10-09T17:11:47.894 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Command line: ntpd -gq 2020-10-09T17:11:47.895 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: proto: precision = 0.123 usec (-23) 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stderr:restrict ::: KOD does nothing without LIMITED. 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: restrict ::: KOD does nothing without LIMITED. 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen and drop on 0 v6wildcard [::]:123 2020-10-09T17:11:47.896 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-10-09T17:11:47.897 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen normally on 2 lo 127.0.0.1:123 2020-10-09T17:11:47.897 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen normally on 3 enp3s0f1 172.21.15.184:123 2020-10-09T17:11:47.897 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen normally on 4 lo [::1]:123 2020-10-09T17:11:47.898 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7b8f%5]:123 2020-10-09T17:11:47.898 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:48 ntpd[8694]: Listening on routing socket on fd #22 for interface updates 2020-10-09T17:11:48.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:49 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:48.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:49 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:48.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:49 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:48.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:49 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:49.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:50 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:49.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:50 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:49.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:50 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:49.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:50 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:50.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:50.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:50.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:50.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:50.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:50.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:50.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:50.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:51 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:51.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:51.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:51.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:51.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:51.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:51.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:51.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:51.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:52 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:52.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:52.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:52.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:52.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:52.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:52.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:52.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:52.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:53 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:53.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:53.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:53.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:53.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:53.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:53.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:53.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:53.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:54.895 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:54.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:54.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:54.896 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:54.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-10-09T17:11:54.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: can't open /var/log/ntpstats/rawstats.20201009: Permission denied 2020-10-09T17:11:54.897 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-10-09T17:11:54.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:55 ntpd[8694]: can't open /var/log/ntpstats/peerstats.20201009: Permission denied 2020-10-09T17:11:54.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-10-09T17:11:54.898 INFO:teuthology.orchestra.run.smithi184.stderr: 9 Oct 17:11:54 ntpd[8694]: can't open /var/log/ntpstats/loopstats.20201009: Permission denied 2020-10-09T17:11:54.899 INFO:teuthology.orchestra.run.smithi184.stdout: 9 Oct 17:11:54 ntpd[8694]: ntpd: time set -0.197008 s 2020-10-09T17:11:54.899 INFO:teuthology.orchestra.run.smithi184.stdout:ntpd: time set -0.197008s 2020-10-09T17:11:54.956 INFO:teuthology.orchestra.run.smithi184.stdout: remote refid st t when poll reach delay offset jitter 2020-10-09T17:11:54.957 INFO:teuthology.orchestra.run.smithi184.stdout:============================================================================== 2020-10-09T17:11:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:54.957 INFO:teuthology.orchestra.run.smithi184.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-10-09T17:11:54.958 INFO:teuthology.run_tasks:Running task install... 2020-10-09T17:11:54.969 DEBUG:teuthology.task.install:project ceph 2020-10-09T17:11:54.970 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'}} 2020-10-09T17:11:54.970 DEBUG:teuthology.task.install:config {'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'} 2020-10-09T17:11:54.970 INFO:teuthology.task.install:Using flavor: basic 2020-10-09T17:11:54.977 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-10-09T17:11:54.978 INFO:teuthology.task.install:extra packages: [] 2020-10-09T17:11:54.978 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-10-09T17:11:55.024 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-10-09T17:11:55.029 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:11:55.030 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-10-09T17:11:55.030 INFO:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2020-10-09T17:11:55.084 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-10-09T17:11:55.089 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:11:55.089 INFO:teuthology.orchestra.run.smithi184:> sudo apt-key list | grep Ceph 2020-10-09T17:11:55.130 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-10-09T17:11:55.181 INFO:teuthology.orchestra.run.smithi184.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-10-09T17:11:55.191 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-10-09T17:11:55.191 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2020-10-09T17:11:55.192 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-10-09T17:11:55.236 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-10-09T17:11:55.236 INFO:teuthology.orchestra.run.smithi184.stdout:uid [ unknown] Ceph.com (release key) 2020-10-09T17:11:55.237 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-10-09T17:11:55.246 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-10-09T17:11:55.251 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:11:55.252 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-10-09T17:11:55.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4b457b3813ec2da916f501ce132355a10e79330a 2020-10-09T17:11:55.288 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-10-09T17:11:55.293 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:11:55.293 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-10-09T17:11:55.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4b457b3813ec2da916f501ce132355a10e79330a 2020-10-09T17:11:55.426 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default/ 2020-10-09T17:11:55.426 INFO:teuthology.task.install.deb:Package version is 16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:11:55.465 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default/ 2020-10-09T17:11:55.466 INFO:teuthology.task.install.deb:Package version is 16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:11:55.575 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:11:55.575 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-09T17:11:55.580 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-09T17:11:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:11:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:11:55.590 INFO:teuthology.orchestra.run.smithi060.stderr:159 bytes copied, 7.6004e-05 s, 2.1 MB/s 2020-10-09T17:11:55.591 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-10-09T17:11:55.598 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:11:55.599 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-09T17:11:55.604 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/apt/sources.list.d/ceph.list 2020-10-09T17:11:55.613 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:11:55.613 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:11:55.613 INFO:teuthology.orchestra.run.smithi184.stderr:159 bytes copied, 6.9717e-05 s, 2.3 MB/s 2020-10-09T17:11:55.614 INFO:teuthology.orchestra.run.smithi184:> sudo apt-get update 2020-10-09T17:11:55.807 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-09T17:11:55.878 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic InRelease 2020-10-09T17:11:55.879 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic InRelease 2020-10-09T17:11:55.903 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-10-09T17:11:55.903 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-10-09T17:11:55.920 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-09T17:11:55.929 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-10-09T17:11:55.930 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-10-09T17:11:55.931 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic Release.gpg 2020-10-09T17:11:55.933 INFO:teuthology.orchestra.run.smithi184.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic Release.gpg 2020-10-09T17:11:55.957 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-10-09T17:11:55.960 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,951 B] 2020-10-09T17:11:55.992 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 Packages [16.1 kB] 2020-10-09T17:11:55.993 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,951 B] 2020-10-09T17:11:55.997 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-09T17:11:56.000 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-10-09T17:11:56.077 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-09T17:11:56.080 INFO:teuthology.orchestra.run.smithi184.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-10-09T17:11:56.399 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 26.7 kB in 1s (45.3 kB/s) 2020-10-09T17:11:56.540 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 26.7 kB in 1s (35.9 kB/s) 2020-10-09T17:11:57.617 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-10-09T17:11:57.664 INFO:teuthology.orchestra.run.smithi184:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6241-g4b457b38-1bionic cephadm=16.0.0-6241-g4b457b38-1bionic ceph-mds=16.0.0-6241-g4b457b38-1bionic ceph-mgr=16.0.0-6241-g4b457b38-1bionic ceph-common=16.0.0-6241-g4b457b38-1bionic ceph-fuse=16.0.0-6241-g4b457b38-1bionic ceph-test=16.0.0-6241-g4b457b38-1bionic radosgw=16.0.0-6241-g4b457b38-1bionic python3-rados=16.0.0-6241-g4b457b38-1bionic python3-rgw=16.0.0-6241-g4b457b38-1bionic python3-cephfs=16.0.0-6241-g4b457b38-1bionic python3-rbd=16.0.0-6241-g4b457b38-1bionic libcephfs2=16.0.0-6241-g4b457b38-1bionic libcephfs-dev=16.0.0-6241-g4b457b38-1bionic librados2=16.0.0-6241-g4b457b38-1bionic librbd1=16.0.0-6241-g4b457b38-1bionic rbd-fuse=16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:11:57.737 INFO:teuthology.orchestra.run.smithi184.stdout:Reading package lists... 2020-10-09T17:11:57.799 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-10-09T17:11:57.848 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-6241-g4b457b38-1bionic cephadm=16.0.0-6241-g4b457b38-1bionic ceph-mds=16.0.0-6241-g4b457b38-1bionic ceph-mgr=16.0.0-6241-g4b457b38-1bionic ceph-common=16.0.0-6241-g4b457b38-1bionic ceph-fuse=16.0.0-6241-g4b457b38-1bionic ceph-test=16.0.0-6241-g4b457b38-1bionic radosgw=16.0.0-6241-g4b457b38-1bionic python3-rados=16.0.0-6241-g4b457b38-1bionic python3-rgw=16.0.0-6241-g4b457b38-1bionic python3-cephfs=16.0.0-6241-g4b457b38-1bionic python3-rbd=16.0.0-6241-g4b457b38-1bionic libcephfs2=16.0.0-6241-g4b457b38-1bionic libcephfs-dev=16.0.0-6241-g4b457b38-1bionic librados2=16.0.0-6241-g4b457b38-1bionic librbd1=16.0.0-6241-g4b457b38-1bionic rbd-fuse=16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:11:57.935 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-10-09T17:11:57.965 INFO:teuthology.orchestra.run.smithi184.stdout:Building dependency tree... 2020-10-09T17:11:57.966 INFO:teuthology.orchestra.run.smithi184.stdout:Reading state information... 2020-10-09T17:11:58.127 INFO:teuthology.orchestra.run.smithi184.stdout:The following additional packages will be installed: 2020-10-09T17:11:58.127 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-10-09T17:11:58.128 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-10-09T17:11:58.129 INFO:teuthology.orchestra.run.smithi184.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-10-09T17:11:58.130 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-09T17:11:58.131 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-10-09T17:11:58.132 INFO:teuthology.orchestra.run.smithi184.stdout:Suggested packages: 2020-10-09T17:11:58.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-10-09T17:11:58.132 INFO:teuthology.orchestra.run.smithi184.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-10-09T17:11:58.132 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-10-09T17:11:58.132 INFO:teuthology.orchestra.run.smithi184.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-10-09T17:11:58.133 INFO:teuthology.orchestra.run.smithi184.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-10-09T17:11:58.134 INFO:teuthology.orchestra.run.smithi184.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-10-09T17:11:58.134 INFO:teuthology.orchestra.run.smithi184.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-10-09T17:11:58.134 INFO:teuthology.orchestra.run.smithi184.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-10-09T17:11:58.134 INFO:teuthology.orchestra.run.smithi184.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-10-09T17:11:58.196 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-10-09T17:11:58.197 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-10-09T17:11:58.231 INFO:teuthology.orchestra.run.smithi184.stdout:The following NEW packages will be installed: 2020-10-09T17:11:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-10-09T17:11:58.231 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-10-09T17:11:58.232 INFO:teuthology.orchestra.run.smithi184.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-10-09T17:11:58.233 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-10-09T17:11:58.234 INFO:teuthology.orchestra.run.smithi184.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-09T17:11:58.235 INFO:teuthology.orchestra.run.smithi184.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-09T17:11:58.236 INFO:teuthology.orchestra.run.smithi184.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-09T17:11:58.236 INFO:teuthology.orchestra.run.smithi184.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-10-09T17:11:58.236 INFO:teuthology.orchestra.run.smithi184.stdout:The following packages will be upgraded: 2020-10-09T17:11:58.236 INFO:teuthology.orchestra.run.smithi184.stdout: librados2 librbd1 2020-10-09T17:11:58.380 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2020-10-09T17:11:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2020-10-09T17:11:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2020-10-09T17:11:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-10-09T17:11:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2020-10-09T17:11:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-10-09T17:11:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-10-09T17:11:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-10-09T17:11:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-10-09T17:11:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-10-09T17:11:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2020-10-09T17:11:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-10-09T17:11:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-10-09T17:11:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-10-09T17:11:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-10-09T17:11:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-10-09T17:11:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-10-09T17:11:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-10-09T17:11:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-09T17:11:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-09T17:11:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-09T17:11:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-09T17:11:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-10-09T17:11:58.386 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2020-10-09T17:11:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-10-09T17:11:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-10-09T17:11:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-10-09T17:11:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-10-09T17:11:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-10-09T17:11:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-10-09T17:11:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-10-09T17:11:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-10-09T17:11:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-10-09T17:11:58.400 INFO:teuthology.orchestra.run.smithi184.stdout:2 upgraded, 110 newly installed, 0 to remove and 7 not upgraded. 2020-10-09T17:11:58.400 INFO:teuthology.orchestra.run.smithi184.stdout:Need to get 167 MB of archives. 2020-10-09T17:11:58.400 INFO:teuthology.orchestra.run.smithi184.stdout:After this operation, 779 MB of additional disk space will be used. 2020-10-09T17:11:58.400 INFO:teuthology.orchestra.run.smithi184.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6241-g4b457b38-1bionic [1,867 kB] 2020-10-09T17:11:58.423 INFO:teuthology.orchestra.run.smithi184.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-10-09T17:11:58.487 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2020-10-09T17:11:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-10-09T17:11:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2020-10-09T17:11:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2020-10-09T17:11:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-dev libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 2020-10-09T17:11:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs python3-cffi-backend python3-cherrypy3 python3-cryptography 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cycler python3-dateutil python3-decorator python3-fasteners 2020-10-09T17:11:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: python3-html5lib python3-httplib2 python3-jinja2 python3-joblib 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-json-pointer python3-jsonpatch python3-jwt python3-kubernetes 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-10-09T17:11:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-10-09T17:11:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-10-09T17:11:58.493 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2020-10-09T17:11:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 librbd1 2020-10-09T17:11:58.503 INFO:teuthology.orchestra.run.smithi184.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-10-09T17:11:58.637 INFO:teuthology.orchestra.run.smithi184.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6241-g4b457b38-1bionic [3,172 kB] 2020-10-09T17:11:58.653 INFO:teuthology.orchestra.run.smithi060.stdout:2 upgraded, 110 newly installed, 0 to remove and 7 not upgraded. 2020-10-09T17:11:58.653 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 167 MB of archives. 2020-10-09T17:11:58.654 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 779 MB of additional disk space will be used. 2020-10-09T17:11:58.654 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-6241-g4b457b38-1bionic [1,867 kB] 2020-10-09T17:11:58.681 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-10-09T17:11:58.756 INFO:teuthology.orchestra.run.smithi184.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-10-09T17:11:58.761 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-10-09T17:11:58.770 INFO:teuthology.orchestra.run.smithi184.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-10-09T17:11:58.788 INFO:teuthology.orchestra.run.smithi184.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-10-09T17:11:58.833 INFO:teuthology.orchestra.run.smithi184.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6241-g4b457b38-1bionic [474 kB] 2020-10-09T17:11:58.854 INFO:teuthology.orchestra.run.smithi184.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6241-g4b457b38-1bionic [327 kB] 2020-10-09T17:11:58.874 INFO:teuthology.orchestra.run.smithi184.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6241-g4b457b38-1bionic [20.6 kB] 2020-10-09T17:11:58.875 INFO:teuthology.orchestra.run.smithi184.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6241-g4b457b38-1bionic [168 kB] 2020-10-09T17:11:58.885 INFO:teuthology.orchestra.run.smithi184.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-10-09T17:11:58.885 INFO:teuthology.orchestra.run.smithi184.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6241-g4b457b38-1bionic [27.7 kB] 2020-10-09T17:11:58.889 INFO:teuthology.orchestra.run.smithi184.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-10-09T17:11:58.889 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-6241-g4b457b38-1bionic [3,172 kB] 2020-10-09T17:11:58.891 INFO:teuthology.orchestra.run.smithi184.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6241-g4b457b38-1bionic [312 kB] 2020-10-09T17:11:58.899 INFO:teuthology.orchestra.run.smithi184.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6241-g4b457b38-1bionic [3,136 kB] 2020-10-09T17:11:58.913 INFO:teuthology.orchestra.run.smithi184.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-10-09T17:11:58.920 INFO:teuthology.orchestra.run.smithi184.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-10-09T17:11:58.929 INFO:teuthology.orchestra.run.smithi184.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-10-09T17:11:58.940 INFO:teuthology.orchestra.run.smithi184.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-10-09T17:11:58.949 INFO:teuthology.orchestra.run.smithi184.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-10-09T17:11:58.973 INFO:teuthology.orchestra.run.smithi184.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-10-09T17:11:58.978 INFO:teuthology.orchestra.run.smithi184.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-10-09T17:11:58.995 INFO:teuthology.orchestra.run.smithi184.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-10-09T17:11:59.006 INFO:teuthology.orchestra.run.smithi184.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-10-09T17:11:59.007 INFO:teuthology.orchestra.run.smithi184.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-10-09T17:11:59.013 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-10-09T17:11:59.027 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-10-09T17:11:59.046 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-10-09T17:11:59.070 INFO:teuthology.orchestra.run.smithi184.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6241-g4b457b38-1bionic [99.2 kB] 2020-10-09T17:11:59.074 INFO:teuthology.orchestra.run.smithi184.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-10-09T17:11:59.074 INFO:teuthology.orchestra.run.smithi184.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6241-g4b457b38-1bionic [413 kB] 2020-10-09T17:11:59.075 INFO:teuthology.orchestra.run.smithi184.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-10-09T17:11:59.076 INFO:teuthology.orchestra.run.smithi184.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-10-09T17:11:59.077 INFO:teuthology.orchestra.run.smithi184.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-10-09T17:11:59.078 INFO:teuthology.orchestra.run.smithi184.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-10-09T17:11:59.080 INFO:teuthology.orchestra.run.smithi184.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-10-09T17:11:59.084 INFO:teuthology.orchestra.run.smithi184.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-10-09T17:11:59.087 INFO:teuthology.orchestra.run.smithi184.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6241-g4b457b38-1bionic [20.3 MB] 2020-10-09T17:11:59.142 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-6241-g4b457b38-1bionic [474 kB] 2020-10-09T17:11:59.142 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-10-09T17:11:59.146 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-10-09T17:11:59.153 INFO:teuthology.orchestra.run.smithi184.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-10-09T17:11:59.153 INFO:teuthology.orchestra.run.smithi184.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-10-09T17:11:59.173 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-10-09T17:11:59.174 INFO:teuthology.orchestra.run.smithi184.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-10-09T17:11:59.181 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-10-09T17:11:59.182 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-6241-g4b457b38-1bionic [327 kB] 2020-10-09T17:11:59.186 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-10-09T17:11:59.198 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-10-09T17:11:59.208 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-10-09T17:11:59.217 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-6241-g4b457b38-1bionic [20.6 kB] 2020-10-09T17:11:59.218 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-6241-g4b457b38-1bionic [168 kB] 2020-10-09T17:11:59.231 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-10-09T17:11:59.232 INFO:teuthology.orchestra.run.smithi184.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-10-09T17:11:59.232 INFO:teuthology.orchestra.run.smithi184.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-10-09T17:11:59.234 INFO:teuthology.orchestra.run.smithi184.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-10-09T17:11:59.235 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-10-09T17:11:59.238 INFO:teuthology.orchestra.run.smithi184.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-10-09T17:11:59.245 INFO:teuthology.orchestra.run.smithi184.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-10-09T17:11:59.248 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-6241-g4b457b38-1bionic [27.7 kB] 2020-10-09T17:11:59.249 INFO:teuthology.orchestra.run.smithi184.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-10-09T17:11:59.249 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-6241-g4b457b38-1bionic [312 kB] 2020-10-09T17:11:59.250 INFO:teuthology.orchestra.run.smithi184.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-10-09T17:11:59.250 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-10-09T17:11:59.261 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-6241-g4b457b38-1bionic [3,136 kB] 2020-10-09T17:11:59.262 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-10-09T17:11:59.311 INFO:teuthology.orchestra.run.smithi184.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-10-09T17:11:59.329 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-10-09T17:11:59.330 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-10-09T17:11:59.331 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-10-09T17:11:59.333 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-10-09T17:11:59.334 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-10-09T17:11:59.335 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-10-09T17:11:59.337 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-10-09T17:11:59.339 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-10-09T17:11:59.397 INFO:teuthology.orchestra.run.smithi184.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-10-09T17:11:59.398 INFO:teuthology.orchestra.run.smithi184.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-10-09T17:11:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-10-09T17:11:59.410 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-10-09T17:11:59.488 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-10-09T17:11:59.489 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-10-09T17:11:59.489 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-10-09T17:11:59.490 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-10-09T17:11:59.494 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-10-09T17:11:59.501 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-10-09T17:11:59.505 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-10-09T17:11:59.506 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-10-09T17:11:59.552 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-6241-g4b457b38-1bionic [99.2 kB] 2020-10-09T17:11:59.555 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-6241-g4b457b38-1bionic [413 kB] 2020-10-09T17:11:59.567 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-10-09T17:11:59.572 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-6241-g4b457b38-1bionic [20.3 MB] 2020-10-09T17:11:59.654 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-10-09T17:11:59.654 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-10-09T17:11:59.670 INFO:teuthology.orchestra.run.smithi184.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-10-09T17:11:59.711 INFO:teuthology.orchestra.run.smithi184.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-10-09T17:11:59.713 INFO:teuthology.orchestra.run.smithi184.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-10-09T17:11:59.751 INFO:teuthology.orchestra.run.smithi184.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-10-09T17:11:59.752 INFO:teuthology.orchestra.run.smithi184.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-10-09T17:11:59.752 INFO:teuthology.orchestra.run.smithi184.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-10-09T17:11:59.752 INFO:teuthology.orchestra.run.smithi184.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-10-09T17:11:59.753 INFO:teuthology.orchestra.run.smithi184.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-10-09T17:11:59.753 INFO:teuthology.orchestra.run.smithi184.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-10-09T17:11:59.754 INFO:teuthology.orchestra.run.smithi184.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-10-09T17:11:59.829 INFO:teuthology.orchestra.run.smithi184.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-10-09T17:11:59.831 INFO:teuthology.orchestra.run.smithi184.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-10-09T17:11:59.909 INFO:teuthology.orchestra.run.smithi184.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-10-09T17:11:59.909 INFO:teuthology.orchestra.run.smithi184.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-10-09T17:11:59.912 INFO:teuthology.orchestra.run.smithi184.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-10-09T17:11:59.912 INFO:teuthology.orchestra.run.smithi184.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-10-09T17:11:59.913 INFO:teuthology.orchestra.run.smithi184.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-10-09T17:11:59.915 INFO:teuthology.orchestra.run.smithi184.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-10-09T17:11:59.917 INFO:teuthology.orchestra.run.smithi184.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-10-09T17:11:59.918 INFO:teuthology.orchestra.run.smithi184.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-10-09T17:11:59.936 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-10-09T17:11:59.973 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-10-09T17:11:59.975 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-10-09T17:11:59.987 INFO:teuthology.orchestra.run.smithi184.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-10-09T17:11:59.993 INFO:teuthology.orchestra.run.smithi184.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-10-09T17:12:00.015 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-10-09T17:12:00.016 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-10-09T17:12:00.016 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-10-09T17:12:00.016 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-10-09T17:12:00.017 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-10-09T17:12:00.018 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-10-09T17:12:00.019 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-10-09T17:12:00.067 INFO:teuthology.orchestra.run.smithi184.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-10-09T17:12:00.073 INFO:teuthology.orchestra.run.smithi184.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-10-09T17:12:00.073 INFO:teuthology.orchestra.run.smithi184.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-10-09T17:12:00.074 INFO:teuthology.orchestra.run.smithi184.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-10-09T17:12:00.074 INFO:teuthology.orchestra.run.smithi184.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-10-09T17:12:00.076 INFO:teuthology.orchestra.run.smithi184.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-10-09T17:12:00.079 INFO:teuthology.orchestra.run.smithi184.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-10-09T17:12:00.080 INFO:teuthology.orchestra.run.smithi184.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-10-09T17:12:00.095 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-10-09T17:12:00.097 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-10-09T17:12:00.104 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-10-09T17:12:00.105 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-10-09T17:12:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-10-09T17:12:00.146 INFO:teuthology.orchestra.run.smithi184.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-10-09T17:12:00.174 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-10-09T17:12:00.175 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-10-09T17:12:00.175 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-10-09T17:12:00.179 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-10-09T17:12:00.182 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-10-09T17:12:00.184 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-10-09T17:12:00.225 INFO:teuthology.orchestra.run.smithi184.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-10-09T17:12:00.239 INFO:teuthology.orchestra.run.smithi184.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6241-g4b457b38-1bionic [5,628 kB] 2020-10-09T17:12:00.245 INFO:teuthology.orchestra.run.smithi184.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-10-09T17:12:00.246 INFO:teuthology.orchestra.run.smithi184.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-10-09T17:12:00.246 INFO:teuthology.orchestra.run.smithi184.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-10-09T17:12:00.253 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-10-09T17:12:00.259 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-10-09T17:12:00.267 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nvme-cli amd64 1.5-1ubuntu1 [183 kB] 2020-10-09T17:12:00.269 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-10-09T17:12:00.333 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-10-09T17:12:00.333 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-10-09T17:12:00.334 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-10-09T17:12:00.334 INFO:teuthology.orchestra.run.smithi184.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-10-09T17:12:00.335 INFO:teuthology.orchestra.run.smithi184.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-10-09T17:12:00.336 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-10-09T17:12:00.339 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-10-09T17:12:00.340 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-10-09T17:12:00.383 INFO:teuthology.orchestra.run.smithi184.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-10-09T17:12:00.391 INFO:teuthology.orchestra.run.smithi184.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-10-09T17:12:00.391 INFO:teuthology.orchestra.run.smithi184.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-10-09T17:12:00.392 INFO:teuthology.orchestra.run.smithi184.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-10-09T17:12:00.393 INFO:teuthology.orchestra.run.smithi184.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-10-09T17:12:00.395 INFO:teuthology.orchestra.run.smithi184.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-10-09T17:12:00.396 INFO:teuthology.orchestra.run.smithi184.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-10-09T17:12:00.412 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-10-09T17:12:00.412 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-10-09T17:12:00.415 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-10-09T17:12:00.436 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-10-09T17:12:00.461 INFO:teuthology.orchestra.run.smithi184.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6241-g4b457b38-1bionic [158 kB] 2020-10-09T17:12:00.465 INFO:teuthology.orchestra.run.smithi184.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6241-g4b457b38-1bionic [1,371 kB] 2020-10-09T17:12:00.493 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-10-09T17:12:00.493 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-10-09T17:12:00.533 INFO:teuthology.orchestra.run.smithi184.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6241-g4b457b38-1bionic [6,647 kB] 2020-10-09T17:12:00.593 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-10-09T17:12:00.593 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-10-09T17:12:00.593 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-10-09T17:12:00.598 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-10-09T17:12:00.598 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-10-09T17:12:00.598 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-10-09T17:12:00.599 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-10-09T17:12:00.600 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-10-09T17:12:00.649 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-10-09T17:12:00.898 INFO:teuthology.orchestra.run.smithi184.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6241-g4b457b38-1bionic [24.6 MB] 2020-10-09T17:12:01.352 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-6241-g4b457b38-1bionic [5,628 kB] 2020-10-09T17:12:01.840 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-6241-g4b457b38-1bionic [158 kB] 2020-10-09T17:12:01.845 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-6241-g4b457b38-1bionic [1,371 kB] 2020-10-09T17:12:01.977 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-6241-g4b457b38-1bionic [6,647 kB] 2020-10-09T17:12:02.187 INFO:teuthology.orchestra.run.smithi184.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6241-g4b457b38-1bionic [3,808 B] 2020-10-09T17:12:02.188 INFO:teuthology.orchestra.run.smithi184.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6241-g4b457b38-1bionic [559 kB] 2020-10-09T17:12:02.230 INFO:teuthology.orchestra.run.smithi184.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6241-g4b457b38-1bionic [1,978 kB] 2020-10-09T17:12:02.322 INFO:teuthology.orchestra.run.smithi184.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6241-g4b457b38-1bionic [51.3 kB] 2020-10-09T17:12:02.324 INFO:teuthology.orchestra.run.smithi184.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6241-g4b457b38-1bionic [55.6 kB] 2020-10-09T17:12:02.325 INFO:teuthology.orchestra.run.smithi184.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6241-g4b457b38-1bionic [2,862 kB] 2020-10-09T17:12:02.477 INFO:teuthology.orchestra.run.smithi184.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6241-g4b457b38-1bionic [7,893 kB] 2020-10-09T17:12:02.518 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-6241-g4b457b38-1bionic [24.6 MB] 2020-10-09T17:12:02.913 INFO:teuthology.orchestra.run.smithi184.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6241-g4b457b38-1bionic [3,928 B] 2020-10-09T17:12:02.914 INFO:teuthology.orchestra.run.smithi184.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6241-g4b457b38-1bionic [22.3 kB] 2020-10-09T17:12:02.914 INFO:teuthology.orchestra.run.smithi184.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6241-g4b457b38-1bionic [49.4 MB] 2020-10-09T17:12:04.595 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-6241-g4b457b38-1bionic [3,808 B] 2020-10-09T17:12:04.596 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-6241-g4b457b38-1bionic [559 kB] 2020-10-09T17:12:04.640 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-6241-g4b457b38-1bionic [1,978 kB] 2020-10-09T17:12:04.791 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-6241-g4b457b38-1bionic [51.3 kB] 2020-10-09T17:12:04.791 INFO:teuthology.orchestra.run.smithi060.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-6241-g4b457b38-1bionic [55.6 kB] 2020-10-09T17:12:04.792 INFO:teuthology.orchestra.run.smithi060.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-6241-g4b457b38-1bionic [2,862 kB] 2020-10-09T17:12:04.990 INFO:teuthology.orchestra.run.smithi060.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-6241-g4b457b38-1bionic [7,893 kB] 2020-10-09T17:12:05.552 INFO:teuthology.orchestra.run.smithi060.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-6241-g4b457b38-1bionic [3,928 B] 2020-10-09T17:12:05.553 INFO:teuthology.orchestra.run.smithi060.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-6241-g4b457b38-1bionic [22.3 kB] 2020-10-09T17:12:05.553 INFO:teuthology.orchestra.run.smithi060.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-6241-g4b457b38-1bionic [49.4 MB] 2020-10-09T17:12:05.706 INFO:teuthology.orchestra.run.smithi184.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6241-g4b457b38-1bionic [19.2 kB] 2020-10-09T17:12:05.707 INFO:teuthology.orchestra.run.smithi184.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6241-g4b457b38-1bionic [9,692 kB] 2020-10-09T17:12:06.298 INFO:teuthology.orchestra.run.smithi184.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6241-g4b457b38-1bionic [76.9 kB] 2020-10-09T17:12:06.991 INFO:teuthology.orchestra.run.smithi184.stdout:Fetched 167 MB in 8s (20.8 MB/s) 2020-10-09T17:12:07.136 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-uritemplate. 2020-10-09T17:12:07.171 INFO:teuthology.orchestra.run.smithi184.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102061 files and directories currently installed.) 2020-10-09T17:12:07.174 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-10-09T17:12:07.218 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-10-09T17:12:07.546 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-10-09T17:12:07.558 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-10-09T17:12:07.576 INFO:teuthology.orchestra.run.smithi060.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-6241-g4b457b38-1bionic [19.2 kB] 2020-10-09T17:12:07.577 INFO:teuthology.orchestra.run.smithi060.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-6241-g4b457b38-1bionic [9,692 kB] 2020-10-09T17:12:07.638 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-09T17:12:07.890 INFO:teuthology.orchestra.run.smithi060.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-investigate-datalog/4b457b3813ec2da916f501ce132355a10e79330a/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-6241-g4b457b38-1bionic [76.9 kB] 2020-10-09T17:12:07.953 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../002-librbd1_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:08.075 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librbd1 (16.0.0-6241-g4b457b38-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-09T17:12:08.593 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 167 MB in 9s (17.9 MB/s) 2020-10-09T17:12:08.624 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../003-librados2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:08.693 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-uritemplate. 2020-10-09T17:12:08.732 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102061 files and directories currently installed.) 2020-10-09T17:12:08.735 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-10-09T17:12:08.737 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librados2 (16.0.0-6241-g4b457b38-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-09T17:12:08.775 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-10-09T17:12:09.094 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-10-09T17:12:09.106 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-10-09T17:12:09.186 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-09T17:12:09.341 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs2. 2020-10-09T17:12:09.353 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:09.391 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:09.476 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-librbd1_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:09.582 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (16.0.0-6241-g4b457b38-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-09T17:12:09.702 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rados. 2020-10-09T17:12:09.715 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:09.752 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rados (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:10.088 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-10-09T17:12:10.101 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:10.130 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-librados2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:10.161 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-argparse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:10.253 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (16.0.0-6241-g4b457b38-1bionic) over (12.2.13-0ubuntu0.18.04.4) ... 2020-10-09T17:12:10.408 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cephfs. 2020-10-09T17:12:10.420 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:10.449 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cephfs (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:10.768 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-ceph-common. 2020-10-09T17:12:10.781 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:10.827 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:10.857 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2020-10-09T17:12:10.869 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:10.899 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:11.138 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-prettytable. 2020-10-09T17:12:11.151 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-10-09T17:12:11.188 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-10-09T17:12:11.201 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2020-10-09T17:12:11.214 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-python3-rados_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:11.251 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:11.418 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rbd. 2020-10-09T17:12:11.429 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:11.458 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rbd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:11.596 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-10-09T17:12:11.608 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:11.652 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:11.786 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-10-09T17:12:11.799 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-10-09T17:12:11.836 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-09T17:12:11.906 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2020-10-09T17:12:11.919 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:11.956 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:12.172 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-10-09T17:12:12.185 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-10-09T17:12:12.231 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-09T17:12:12.234 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2020-10-09T17:12:12.247 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:12.284 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:12.525 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package librgw2. 2020-10-09T17:12:12.536 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../013-librgw2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:12.581 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking librgw2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:12.588 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2020-10-09T17:12:12.601 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-10-09T17:12:12.639 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-10-09T17:12:12.891 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2020-10-09T17:12:12.903 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:12.941 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:13.094 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rgw. 2020-10-09T17:12:13.107 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:13.144 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:13.277 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-10-09T17:12:13.288 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-10-09T17:12:13.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-09T17:12:13.463 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-10-09T17:12:13.474 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-10-09T17:12:13.513 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-09T17:12:13.655 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-10-09T17:12:13.666 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-10-09T17:12:13.705 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-09T17:12:13.816 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-10-09T17:12:13.828 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-10-09T17:12:13.866 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-10-09T17:12:14.035 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2020-10-09T17:12:14.046 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-librgw2_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:14.076 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:14.177 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package liboath0. 2020-10-09T17:12:14.190 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-10-09T17:12:14.219 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-10-09T17:12:14.538 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libradosstriper1. 2020-10-09T17:12:14.550 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:14.589 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libradosstriper1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:14.620 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2020-10-09T17:12:14.631 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:14.670 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:15.023 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-10-09T17:12:15.036 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-10-09T17:12:15.041 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-common. 2020-10-09T17:12:15.053 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:15.073 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-09T17:12:15.092 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:15.367 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-10-09T17:12:15.379 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-10-09T17:12:15.417 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-10-09T17:12:15.737 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0. 2020-10-09T17:12:15.748 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-10-09T17:12:15.779 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-10-09T17:12:16.081 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2020-10-09T17:12:16.093 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:16.131 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:16.593 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2020-10-09T17:12:16.604 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-ceph-common_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:16.651 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:16.966 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-base. 2020-10-09T17:12:16.976 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:17.024 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-base (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:17.940 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-dateutil. 2020-10-09T17:12:17.953 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-10-09T17:12:17.990 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-10-09T17:12:18.309 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-asn1crypto. 2020-10-09T17:12:18.322 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-10-09T17:12:18.359 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-10-09T17:12:18.532 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2020-10-09T17:12:18.544 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-ceph-base_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:18.591 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:18.662 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cffi-backend. 2020-10-09T17:12:18.674 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-10-09T17:12:18.721 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-10-09T17:12:18.982 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cryptography. 2020-10-09T17:12:18.995 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-10-09T17:12:19.032 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-09T17:12:19.362 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-openssl. 2020-10-09T17:12:19.375 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-10-09T17:12:19.412 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-10-09T17:12:19.506 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-dateutil. 2020-10-09T17:12:19.517 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-10-09T17:12:19.555 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-10-09T17:12:19.664 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-10-09T17:12:19.677 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:19.715 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:19.883 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-asn1crypto. 2020-10-09T17:12:19.896 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-10-09T17:12:19.934 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-10-09T17:12:20.038 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bcrypt. 2020-10-09T17:12:20.051 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-10-09T17:12:20.088 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-10-09T17:12:20.212 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cffi-backend. 2020-10-09T17:12:20.223 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-10-09T17:12:20.262 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-10-09T17:12:20.408 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cherrypy3. 2020-10-09T17:12:20.419 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-10-09T17:12:20.458 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-10-09T17:12:20.540 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cryptography. 2020-10-09T17:12:20.550 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-10-09T17:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-09T17:12:20.794 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jwt. 2020-10-09T17:12:20.807 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-10-09T17:12:20.844 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-10-09T17:12:20.944 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2020-10-09T17:12:20.956 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-10-09T17:12:20.995 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-10-09T17:12:21.156 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-logutils. 2020-10-09T17:12:21.168 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-10-09T17:12:21.206 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-10-09T17:12:21.240 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-10-09T17:12:21.253 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:21.281 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:21.475 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-markupsafe. 2020-10-09T17:12:21.488 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-10-09T17:12:21.517 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-10-09T17:12:21.811 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-mako. 2020-10-09T17:12:21.823 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-10-09T17:12:21.861 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-10-09T17:12:21.887 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2020-10-09T17:12:21.900 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-10-09T17:12:21.929 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-10-09T17:12:22.190 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplegeneric. 2020-10-09T17:12:22.201 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-10-09T17:12:22.239 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-10-09T17:12:22.332 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2020-10-09T17:12:22.345 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-10-09T17:12:22.382 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-10-09T17:12:22.710 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jwt. 2020-10-09T17:12:22.723 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-10-09T17:12:22.769 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-10-09T17:12:22.884 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-singledispatch. 2020-10-09T17:12:22.897 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-10-09T17:12:22.936 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-10-09T17:12:23.105 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2020-10-09T17:12:23.118 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-10-09T17:12:23.155 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-10-09T17:12:23.211 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webob. 2020-10-09T17:12:23.224 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-10-09T17:12:23.261 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-10-09T17:12:23.457 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2020-10-09T17:12:23.470 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-10-09T17:12:23.507 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-10-09T17:12:23.539 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-bs4. 2020-10-09T17:12:23.552 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-10-09T17:12:23.581 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-10-09T17:12:23.802 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2020-10-09T17:12:23.815 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-10-09T17:12:23.844 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-10-09T17:12:23.892 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-waitress. 2020-10-09T17:12:23.905 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-10-09T17:12:23.942 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-10-09T17:12:24.138 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2020-10-09T17:12:24.150 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-10-09T17:12:24.188 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-10-09T17:12:24.262 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tempita. 2020-10-09T17:12:24.272 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-10-09T17:12:24.312 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-10-09T17:12:24.491 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2020-10-09T17:12:24.503 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-10-09T17:12:24.541 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-10-09T17:12:24.615 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-paste. 2020-10-09T17:12:24.626 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-10-09T17:12:24.665 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-09T17:12:24.869 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2020-10-09T17:12:25.041 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-10-09T17:12:25.041 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-10-09T17:12:25.042 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-10-09T17:12:25.042 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-10-09T17:12:25.042 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-10-09T17:12:25.238 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bs4. 2020-10-09T17:12:25.251 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-10-09T17:12:25.288 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-10-09T17:12:25.313 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastedeploy. 2020-10-09T17:12:25.325 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-10-09T17:12:25.372 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-10-09T17:12:25.558 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2020-10-09T17:12:25.570 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-10-09T17:12:25.608 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-10-09T17:12:25.661 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webtest. 2020-10-09T17:12:25.662 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-10-09T17:12:25.677 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-10-09T17:12:25.911 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2020-10-09T17:12:25.923 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-10-09T17:12:25.961 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-10-09T17:12:26.022 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pecan. 2020-10-09T17:12:26.034 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-10-09T17:12:26.072 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-10-09T17:12:26.264 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2020-10-09T17:12:26.276 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-10-09T17:12:26.314 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-09T17:12:26.402 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-werkzeug. 2020-10-09T17:12:26.415 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-10-09T17:12:26.452 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-09T17:12:26.626 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-10-09T17:12:26.637 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-10-09T17:12:26.785 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-10-09T17:12:26.813 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr. 2020-10-09T17:12:26.825 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:26.863 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:27.057 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2020-10-09T17:12:27.069 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-10-09T17:12:27.099 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-10-09T17:12:27.308 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mon. 2020-10-09T17:12:27.321 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:27.350 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mon (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:27.393 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2020-10-09T17:12:27.405 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-10-09T17:12:27.443 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-10-09T17:12:27.780 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2020-10-09T17:12:27.792 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-10-09T17:12:27.830 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-10-09T17:12:28.159 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2020-10-09T17:12:28.170 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-10-09T17:12:28.201 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-09T17:12:28.521 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2020-10-09T17:12:28.532 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:28.563 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:28.645 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-osd. 2020-10-09T17:12:28.657 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:28.695 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-osd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:28.974 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2020-10-09T17:12:28.985 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:29.015 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:29.911 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2020-10-09T17:12:29.924 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:29.960 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:30.675 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph. 2020-10-09T17:12:30.687 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../048-ceph_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:30.717 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:31.046 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-fuse. 2020-10-09T17:12:31.057 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:31.096 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:31.450 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mds. 2020-10-09T17:12:31.463 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:31.499 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mds (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:32.002 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package cephadm. 2020-10-09T17:12:32.015 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../051-cephadm_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:32.052 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:32.363 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jinja2. 2020-10-09T17:12:32.376 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-10-09T17:12:32.414 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-09T17:12:32.609 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2020-10-09T17:12:32.620 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-ceph_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:32.650 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:32.945 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2020-10-09T17:12:32.956 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:32.995 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:33.067 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-10-09T17:12:33.078 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:33.108 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:33.348 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2020-10-09T17:12:33.360 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:33.390 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:33.420 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-repoze.lru. 2020-10-09T17:12:33.432 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-10-09T17:12:33.470 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-10-09T17:12:33.773 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-routes. 2020-10-09T17:12:33.785 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-10-09T17:12:33.823 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-10-09T17:12:33.918 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2020-10-09T17:12:33.931 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-cephadm_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:33.959 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:34.118 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-10-09T17:12:34.130 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:34.168 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:34.279 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jinja2. 2020-10-09T17:12:34.291 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-10-09T17:12:34.329 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-09T17:12:34.649 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-10-09T17:12:34.660 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:34.690 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:34.962 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-numpy. 2020-10-09T17:12:34.976 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-10-09T17:12:35.018 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2020-10-09T17:12:35.029 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-09T17:12:35.031 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-10-09T17:12:35.077 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-10-09T17:12:35.398 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2020-10-09T17:12:35.409 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-10-09T17:12:35.447 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-10-09T17:12:35.668 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-decorator. 2020-10-09T17:12:35.681 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-10-09T17:12:35.718 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-10-09T17:12:35.756 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-10-09T17:12:35.768 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:35.806 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:35.979 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-scipy. 2020-10-09T17:12:35.990 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-10-09T17:12:36.020 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-10-09T17:12:36.684 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-numpy. 2020-10-09T17:12:36.697 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-10-09T17:12:36.734 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-09T17:12:37.169 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-10-09T17:12:37.183 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-10-09T17:12:37.228 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-10-09T17:12:37.331 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-decorator. 2020-10-09T17:12:37.344 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-10-09T17:12:37.381 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-10-09T17:12:37.673 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-joblib. 2020-10-09T17:12:37.684 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-10-09T17:12:37.692 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-scipy. 2020-10-09T17:12:37.705 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-10-09T17:12:37.722 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-joblib (0.11-1) ... 2020-10-09T17:12:37.742 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-10-09T17:12:38.110 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sklearn. 2020-10-09T17:12:38.122 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-10-09T17:12:38.161 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-10-09T17:12:38.694 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-10-09T17:12:38.705 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:38.744 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:38.900 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-10-09T17:12:38.914 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-10-09T17:12:38.959 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-10-09T17:12:39.413 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2020-10-09T17:12:39.425 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-10-09T17:12:39.463 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.11-1) ... 2020-10-09T17:12:39.844 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2020-10-09T17:12:39.858 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-10-09T17:12:39.880 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-monotonic. 2020-10-09T17:12:39.893 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-10-09T17:12:39.902 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-10-09T17:12:39.930 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-10-09T17:12:40.191 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-fasteners. 2020-10-09T17:12:40.205 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-10-09T17:12:40.241 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-10-09T17:12:40.433 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-10-09T17:12:40.445 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:40.484 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:40.544 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-httplib2. 2020-10-09T17:12:40.555 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-10-09T17:12:40.594 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-09T17:12:40.905 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1. 2020-10-09T17:12:40.917 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-10-09T17:12:40.955 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-10-09T17:12:41.283 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-10-09T17:12:41.294 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-10-09T17:12:41.333 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-09T17:12:41.620 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-rsa. 2020-10-09T17:12:41.634 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-10-09T17:12:41.661 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-monotonic. 2020-10-09T17:12:41.674 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-10-09T17:12:41.684 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-10-09T17:12:41.703 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-10-09T17:12:42.006 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-fasteners. 2020-10-09T17:12:42.011 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-10-09T17:12:42.019 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-10-09T17:12:42.024 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-10-09T17:12:42.061 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-09T17:12:42.089 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-10-09T17:12:42.375 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-httplib2. 2020-10-09T17:12:42.389 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-10-09T17:12:42.433 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-09T17:12:42.474 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-oauth2client. 2020-10-09T17:12:42.485 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-10-09T17:12:42.524 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-10-09T17:12:42.745 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2020-10-09T17:12:42.758 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-10-09T17:12:42.795 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-10-09T17:12:42.827 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-setuptools. 2020-10-09T17:12:42.839 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-10-09T17:12:42.869 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-10-09T17:12:43.158 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-10-09T17:12:43.171 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-10-09T17:12:43.208 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-09T17:12:43.256 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-websocket. 2020-10-09T17:12:43.269 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-10-09T17:12:43.306 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-10-09T17:12:43.507 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2020-10-09T17:12:43.521 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-10-09T17:12:43.565 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-10-09T17:12:43.603 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-kubernetes. 2020-10-09T17:12:43.615 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-10-09T17:12:43.660 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-09T17:12:43.835 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-10-09T17:12:43.848 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-10-09T17:12:43.877 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-09T17:12:44.106 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-10-09T17:12:44.119 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:44.164 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:44.332 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-oauth2client. 2020-10-09T17:12:44.345 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-10-09T17:12:44.382 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-10-09T17:12:44.441 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-json-pointer. 2020-10-09T17:12:44.454 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-10-09T17:12:44.491 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-10-09T17:12:44.718 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-setuptools. 2020-10-09T17:12:44.731 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-10-09T17:12:44.790 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-10-09T17:12:44.844 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-jsonpatch. 2020-10-09T17:12:44.857 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-10-09T17:12:44.897 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-09T17:12:45.180 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-10-09T17:12:45.181 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2020-10-09T17:12:45.193 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-10-09T17:12:45.194 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:45.231 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-10-09T17:12:45.238 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-mgr-rook (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:45.518 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2020-10-09T17:12:45.530 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-10-09T17:12:45.550 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libonig4:amd64. 2020-10-09T17:12:45.563 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-10-09T17:12:45.567 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-09T17:12:45.600 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-10-09T17:12:45.944 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libjq1:amd64. 2020-10-09T17:12:45.958 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-10-09T17:12:45.963 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-10-09T17:12:45.974 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:46.013 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:46.079 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-10-09T17:12:46.332 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-json-pointer. 2020-10-09T17:12:46.345 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-10-09T17:12:46.348 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package jq. 2020-10-09T17:12:46.361 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-10-09T17:12:46.382 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-10-09T17:12:46.398 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-10-09T17:12:46.646 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package socat. 2020-10-09T17:12:46.659 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-10-09T17:12:46.696 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-10-09T17:12:46.752 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jsonpatch. 2020-10-09T17:12:46.765 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-10-09T17:12:46.805 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-09T17:12:47.058 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-10-09T17:12:47.069 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-10-09T17:12:47.080 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-10-09T17:12:47.094 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-6241-g4b457b38-1bionic_all.deb ... 2020-10-09T17:12:47.107 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-09T17:12:47.130 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-rook (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:47.419 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package xmlstarlet. 2020-10-09T17:12:47.431 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-10-09T17:12:47.458 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig4:amd64. 2020-10-09T17:12:47.468 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-10-09T17:12:47.470 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-10-09T17:12:47.500 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-10-09T17:12:47.815 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package ceph-test. 2020-10-09T17:12:47.827 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:47.844 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2020-10-09T17:12:47.857 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-10-09T17:12:47.864 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking ceph-test (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:47.971 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-10-09T17:12:48.261 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2020-10-09T17:12:48.273 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-10-09T17:12:48.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-10-09T17:12:48.572 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2020-10-09T17:12:48.584 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-10-09T17:12:48.622 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-10-09T17:12:49.009 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-10-09T17:12:49.021 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-10-09T17:12:49.059 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-09T17:12:49.311 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2020-10-09T17:12:49.323 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-10-09T17:12:49.361 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-10-09T17:12:49.682 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2020-10-09T17:12:49.696 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-ceph-test_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:49.740 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:51.451 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package libcephfs-dev. 2020-10-09T17:12:51.464 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:51.509 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking libcephfs-dev (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:51.761 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package nvme-cli. 2020-10-09T17:12:51.774 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-10-09T17:12:51.811 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-10-09T17:12:52.157 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-attr. 2020-10-09T17:12:52.170 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-10-09T17:12:52.207 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-attr (17.4.0-2) ... 2020-10-09T17:12:52.518 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-funcsigs. 2020-10-09T17:12:52.530 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-10-09T17:12:52.560 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-10-09T17:12:52.838 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pluggy. 2020-10-09T17:12:52.851 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-10-09T17:12:52.896 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-10-09T17:12:53.440 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-py. 2020-10-09T17:12:53.443 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2020-10-09T17:12:53.453 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-10-09T17:12:53.457 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:12:53.490 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-py (1.5.2-1) ... 2020-10-09T17:12:53.493 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:12:53.754 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package nvme-cli. 2020-10-09T17:12:53.766 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-nvme-cli_1.5-1ubuntu1_amd64.deb ... 2020-10-09T17:12:53.802 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python-pytest. 2020-10-09T17:12:53.804 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking nvme-cli (1.5-1ubuntu1) ... 2020-10-09T17:12:53.815 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-10-09T17:12:53.860 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-10-09T17:12:54.109 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-attr. 2020-10-09T17:12:54.123 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-10-09T17:12:54.151 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-attr (17.4.0-2) ... 2020-10-09T17:12:54.197 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-attr. 2020-10-09T17:12:54.210 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-10-09T17:12:54.247 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-10-09T17:12:54.379 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-funcsigs. 2020-10-09T17:12:54.392 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-10-09T17:12:54.429 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-10-09T17:12:54.549 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-cycler. 2020-10-09T17:12:54.562 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-10-09T17:12:54.599 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-10-09T17:12:54.731 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pluggy. 2020-10-09T17:12:54.743 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-10-09T17:12:54.781 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-10-09T17:12:54.868 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-webencodings. 2020-10-09T17:12:54.880 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-10-09T17:12:54.910 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-10-09T17:12:55.067 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-py. 2020-10-09T17:12:55.080 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-10-09T17:12:55.109 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-py (1.5.2-1) ... 2020-10-09T17:12:55.213 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-html5lib. 2020-10-09T17:12:55.226 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-10-09T17:12:55.263 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-10-09T17:12:55.361 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pytest. 2020-10-09T17:12:55.375 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-10-09T17:12:55.420 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-10-09T17:12:55.601 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-10-09T17:12:55.615 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-10-09T17:12:55.659 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-09T17:12:55.707 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-attr. 2020-10-09T17:12:55.718 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-10-09T17:12:55.748 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-10-09T17:12:56.034 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cycler. 2020-10-09T17:12:56.048 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-10-09T17:12:56.084 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-10-09T17:12:56.104 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyparsing. 2020-10-09T17:12:56.117 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-10-09T17:12:56.154 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-09T17:12:56.363 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webencodings. 2020-10-09T17:12:56.377 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-10-09T17:12:56.421 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-10-09T17:12:56.424 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tz. 2020-10-09T17:12:56.439 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-10-09T17:12:56.483 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tz (2018.3-2) ... 2020-10-09T17:12:56.725 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-html5lib. 2020-10-09T17:12:56.737 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-10-09T17:12:56.767 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-10-09T17:12:56.786 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-matplotlib. 2020-10-09T17:12:56.798 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-10-09T17:12:56.840 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-09T17:12:57.078 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-10-09T17:12:57.092 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-10-09T17:12:57.128 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-09T17:12:57.513 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-olefile. 2020-10-09T17:12:57.526 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-10-09T17:12:57.533 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyparsing. 2020-10-09T17:12:57.545 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-10-09T17:12:57.563 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-10-09T17:12:57.574 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-09T17:12:57.877 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pastescript. 2020-10-09T17:12:57.889 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-10-09T17:12:57.890 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tz. 2020-10-09T17:12:57.903 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-10-09T17:12:57.927 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-10-09T17:12:57.939 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tz (2018.3-2) ... 2020-10-09T17:12:58.243 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-matplotlib. 2020-10-09T17:12:58.255 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-10-09T17:12:58.280 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pil:amd64. 2020-10-09T17:12:58.284 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-09T17:12:58.292 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-10-09T17:12:58.330 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-09T17:12:58.692 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pluggy. 2020-10-09T17:12:58.705 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-10-09T17:12:58.742 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-10-09T17:12:58.961 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-olefile. 2020-10-09T17:12:58.974 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-10-09T17:12:59.011 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-10-09T17:12:59.028 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-py. 2020-10-09T17:12:59.039 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-10-09T17:12:59.078 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-py (1.5.2-1) ... 2020-10-09T17:12:59.314 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2020-10-09T17:12:59.329 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-10-09T17:12:59.373 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-10-09T17:12:59.398 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pyinotify. 2020-10-09T17:12:59.410 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-10-09T17:12:59.456 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-10-09T17:12:59.725 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-pytest. 2020-10-09T17:12:59.734 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pil:amd64. 2020-10-09T17:12:59.737 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-10-09T17:12:59.749 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-10-09T17:12:59.775 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-10-09T17:12:59.792 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-09T17:13:00.087 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-simplejson. 2020-10-09T17:13:00.101 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-10-09T17:13:00.137 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-10-09T17:13:00.162 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2020-10-09T17:13:00.177 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-10-09T17:13:00.221 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-10-09T17:13:00.431 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package python3-tk:amd64. 2020-10-09T17:13:00.446 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-10-09T17:13:00.498 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-09T17:13:00.507 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2020-10-09T17:13:00.521 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-10-09T17:13:00.565 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.5.2-1) ... 2020-10-09T17:13:00.893 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2020-10-09T17:13:00.905 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-10-09T17:13:00.943 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-10-09T17:13:00.985 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package radosgw. 2020-10-09T17:13:00.997 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../110-radosgw_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:13:01.035 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking radosgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:01.204 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2020-10-09T17:13:01.216 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-10-09T17:13:01.254 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-10-09T17:13:01.618 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2020-10-09T17:13:01.629 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-10-09T17:13:01.668 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-10-09T17:13:02.012 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tk:amd64. 2020-10-09T17:13:02.025 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-10-09T17:13:02.046 INFO:teuthology.orchestra.run.smithi184.stdout:Selecting previously unselected package rbd-fuse. 2020-10-09T17:13:02.060 INFO:teuthology.orchestra.run.smithi184.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:13:02.062 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-09T17:13:02.104 INFO:teuthology.orchestra.run.smithi184.stdout:Unpacking rbd-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:02.399 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-10-09T17:13:02.508 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2020-10-09T17:13:02.522 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../110-radosgw_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:13:02.566 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:02.701 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-09T17:13:03.061 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-09T17:13:03.187 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-09T17:13:03.527 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2020-10-09T17:13:03.539 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../111-rbd-fuse_16.0.0-6241-g4b457b38-1bionic_amd64.deb ... 2020-10-09T17:13:03.569 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:03.797 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-09T17:13:03.871 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-10-09T17:13:04.157 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-10-09T17:13:04.182 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-10-09T17:13:04.442 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webencodings (0.5-2) ... 2020-10-09T17:13:04.543 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-10-09T17:13:04.652 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-10-09T17:13:04.734 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-attr (17.4.0-2) ... 2020-10-09T17:13:04.919 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-10-09T17:13:05.020 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-py (1.5.2-1) ... 2020-10-09T17:13:05.263 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-10-09T17:13:05.338 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-10-09T17:13:05.647 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-10-09T17:13:05.724 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-10-09T17:13:05.940 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webencodings (0.5-2) ... 2020-10-09T17:13:06.018 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-10-09T17:13:06.242 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-attr (17.4.0-2) ... 2020-10-09T17:13:06.302 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-argparse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:06.443 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-10-09T17:13:06.578 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-10-09T17:13:06.585 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.5.2-1) ... 2020-10-09T17:13:06.838 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-10-09T17:13:06.920 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-10-09T17:13:06.938 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-09T17:13:07.332 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-10-09T17:13:07.640 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-10-09T17:13:07.858 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-10-09T17:13:07.950 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:08.177 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-10-09T17:13:08.259 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-10-09T17:13:08.436 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-py (1.5.2-1) ... 2020-10-09T17:13:08.569 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-10-09T17:13:08.654 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-10-09T17:13:08.687 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-10-09T17:13:08.955 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-10-09T17:13:09.031 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-10-09T17:13:09.257 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-10-09T17:13:09.416 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-10-09T17:13:09.617 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-10-09T17:13:09.632 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-10-09T17:13:09.927 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-09T17:13:09.950 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-10-09T17:13:10.226 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-py (1.5.2-1) ... 2020-10-09T17:13:10.235 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-10-09T17:13:10.461 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-10-09T17:13:10.503 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-10-09T17:13:10.812 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-10-09T17:13:10.838 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-10-09T17:13:10.901 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-10-09T17:13:11.139 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-10-09T17:13:11.146 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-10-09T17:13:11.273 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-10-09T17:13:11.415 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:11.474 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-10-09T17:13:11.792 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-10-09T17:13:12.107 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user cephadm....done 2020-10-09T17:13:12.118 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-10-09T17:13:12.230 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-10-09T17:13:12.402 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-10-09T17:13:12.506 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-09T17:13:12.607 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-10-09T17:13:12.733 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-10-09T17:13:12.779 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-10-09T17:13:13.076 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-10-09T17:13:13.113 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-10-09T17:13:13.377 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-10-09T17:13:13.448 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:13.495 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-10-09T17:13:13.788 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:14.064 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-monotonic (1.1-2) ... 2020-10-09T17:13:14.099 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2020-10-09T17:13:14.263 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-10-09T17:13:14.331 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-09T17:13:14.432 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-09T17:13:14.556 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-10-09T17:13:14.690 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up nvme-cli (1.5-1ubuntu1) ... 2020-10-09T17:13:14.783 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-10-09T17:13:14.816 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-10-09T17:13:15.135 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-09T17:13:15.201 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-10-09T17:13:15.224 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-10-09T17:13:15.265 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-10-09T17:13:15.487 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-10-09T17:13:15.502 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-10-09T17:13:15.611 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-10-09T17:13:15.771 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-09T17:13:15.946 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:16.055 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-09T17:13:16.272 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-monotonic (1.1-2) ... 2020-10-09T17:13:16.374 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-attr (17.4.0-2) ... 2020-10-09T17:13:16.574 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-10-09T17:13:16.658 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-10-09T17:13:16.700 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-10-09T17:13:16.851 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-09T17:13:17.051 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-10-09T17:13:17.252 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-10-09T17:13:17.342 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-10-09T17:13:17.411 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-10-09T17:13:17.484 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-10-09T17:13:17.524 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-10-09T17:13:17.595 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liboath0 (2.6.1-1) ... 2020-10-09T17:13:17.762 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-10-09T17:13:17.771 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-09T17:13:17.872 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-09T17:13:18.072 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-10-09T17:13:18.115 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-10-09T17:13:18.233 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-10-09T17:13:18.373 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-10-09T17:13:18.626 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-09T17:13:18.724 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-attr (17.4.0-2) ... 2020-10-09T17:13:18.752 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-10-09T17:13:19.042 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-10-09T17:13:19.044 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-tz (2018.3-2) ... 2020-10-09T17:13:19.243 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-10-09T17:13:19.312 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-10-09T17:13:19.572 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-10-09T17:13:19.662 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-10-09T17:13:19.743 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-10-09T17:13:19.856 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-joblib (0.11-1) ... 2020-10-09T17:13:19.997 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0 (2.6.1-1) ... 2020-10-09T17:13:20.173 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-10-09T17:13:20.192 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-routes (2.4.1-1) ... 2020-10-09T17:13:20.308 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-10-09T17:13:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-10-09T17:13:20.567 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-10-09T17:13:20.693 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-10-09T17:13:20.785 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-10-09T17:13:21.071 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-10-09T17:13:21.103 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-09T17:13:21.171 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-10-09T17:13:21.454 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-10-09T17:13:21.455 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tz (2018.3-2) ... 2020-10-09T17:13:21.722 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up jq (1.5+dfsg-2) ... 2020-10-09T17:13:21.773 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-10-09T17:13:21.823 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-10-09T17:13:22.082 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-10-09T17:13:22.392 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.11-1) ... 2020-10-09T17:13:22.769 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.4.1-1) ... 2020-10-09T17:13:22.979 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-09T17:13:23.095 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-10-09T17:13:23.280 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-10-09T17:13:23.404 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-10-09T17:13:23.589 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-10-09T17:13:23.690 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-10-09T17:13:23.714 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-10-09T17:13:24.091 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-10-09T17:13:24.217 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python-pytest (3.3.2-2) ... 2020-10-09T17:13:24.409 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.5+dfsg-2) ... 2020-10-09T17:13:24.476 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-10-09T17:13:24.535 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-10-09T17:13:24.602 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-10-09T17:13:24.720 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librados2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:24.821 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-10-09T17:13:25.097 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:25.206 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libcephfs-dev (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:25.316 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rados (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:25.416 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-10-09T17:13:25.726 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-10-09T17:13:25.749 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-10-09T17:13:26.027 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-10-09T17:13:26.067 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-10-09T17:13:26.362 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-09T17:13:26.393 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-10-09T17:13:26.536 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-10-09T17:13:26.904 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pytest (3.3.2-2) ... 2020-10-09T17:13:27.139 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-10-09T17:13:27.265 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-10-09T17:13:27.274 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-10-09T17:13:27.399 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:27.525 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-10-09T17:13:27.835 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:27.960 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:28.088 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-10-09T17:13:28.216 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:28.364 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up libradosstriper1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:28.465 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-cephfs (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:28.496 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-10-09T17:13:28.733 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librgw2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:28.805 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-10-09T17:13:28.842 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:29.057 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-09T17:13:29.058 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-09T17:13:29.115 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-10-09T17:13:29.327 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up librbd1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:29.452 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-10-09T17:13:29.466 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-10-09T17:13:29.542 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-10-09T17:13:29.575 INFO:teuthology.orchestra.run.smithi184.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-10-09T17:13:29.872 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-09T17:13:30.429 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-10-09T17:13:30.508 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-modules-core (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:30.634 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-10-09T17:13:30.944 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.069 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up rbd-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.187 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up python3-rbd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.267 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-10-09T17:13:31.288 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.568 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.694 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:31.750 INFO:teuthology.orchestra.run.smithi184.stdout:Adding group ceph....done 2020-10-09T17:13:31.987 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:32.088 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:32.150 INFO:teuthology.orchestra.run.smithi184.stdout:Adding system user ceph....done 2020-10-09T17:13:32.308 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-09T17:13:32.308 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-10-09T17:13:32.309 INFO:teuthology.orchestra.run.smithi184.stdout:Setting system user ceph properties....done 2020-10-09T17:13:32.460 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-10-09T17:13:32.598 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:32.688 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-10-09T17:13:32.724 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-10-09T17:13:32.814 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-10-09T17:13:32.846 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-10-09T17:13:32.981 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up radosgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:33.169 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-10-09T17:13:33.222 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-09T17:13:33.223 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-09T17:13:33.633 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-test (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:33.759 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-base (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:33.863 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:33.981 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-10-09T17:13:34.066 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-10-09T17:13:34.299 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:34.378 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:34.433 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:34.559 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:34.593 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-09T17:13:34.594 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-09T17:13:34.685 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:34.896 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:35.013 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-osd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:35.168 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2020-10-09T17:13:35.416 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-09T17:13:35.417 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-09T17:13:35.564 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2020-10-09T17:13:35.699 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:35.764 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2020-10-09T17:13:35.825 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-rook (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:35.927 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-10-09T17:13:35.951 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mds (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:36.150 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-10-09T17:13:36.174 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-09T17:13:36.175 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-09T17:13:36.437 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:36.469 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mon (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:36.690 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-09T17:13:36.690 INFO:teuthology.orchestra.run.smithi184.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-09T17:13:36.702 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-09T17:13:36.702 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-10-09T17:13:36.953 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:37.071 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph-mgr-dashboard (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:37.088 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:37.189 INFO:teuthology.orchestra.run.smithi184.stdout:Setting up ceph (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:37.190 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:37.290 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-10-09T17:13:37.491 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-10-09T17:13:37.511 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-10-09T17:13:37.791 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:38.019 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-09T17:13:38.020 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-10-09T17:13:38.318 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:38.686 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:39.130 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-09T17:13:39.130 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-10-09T17:13:39.397 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:39.515 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-rook (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:39.545 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-10-09T17:13:39.593 INFO:teuthology.orchestra.run.smithi184.stdout:ureadahead will be reprofiled on next reboot 2020-10-09T17:13:39.632 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:39.653 INFO:teuthology.orchestra.run.smithi184.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-10-09T17:13:39.844 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-09T17:13:39.845 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-10-09T17:13:40.093 INFO:teuthology.orchestra.run.smithi184.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-10-09T17:13:40.096 DEBUG:teuthology.parallel:result is None 2020-10-09T17:13:40.150 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:40.372 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-09T17:13:40.372 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-10-09T17:13:40.643 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:40.769 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:40.912 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (16.0.0-6241-g4b457b38-1bionic) ... 2020-10-09T17:13:41.013 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-10-09T17:13:41.198 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-10-09T17:13:43.301 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-10-09T17:13:43.347 INFO:teuthology.orchestra.run.smithi060.stdout:ureadahead will be reprofiled on next reboot 2020-10-09T17:13:43.402 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-10-09T17:13:43.897 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-10-09T17:13:43.899 DEBUG:teuthology.parallel:result is None 2020-10-09T17:13:43.899 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-10-09T17:13:43.954 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-10-09T17:13:43.959 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:13:43.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4b457b3813ec2da916f501ce132355a10e79330a 2020-10-09T17:13:44.132 INFO:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2020-10-09T17:13:44.161 INFO:teuthology.orchestra.run.smithi060.stdout:16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:13:44.162 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:13:44.163 INFO:teuthology.task.install:The correct ceph version 16.0.0-6241-g4b457b38-1bionic is installed. 2020-10-09T17:13:44.163 INFO:teuthology.orchestra.run.smithi184:> sudo lsb_release -is 2020-10-09T17:13:44.221 INFO:teuthology.orchestra.run.smithi184.stdout:Ubuntu 2020-10-09T17:13:44.226 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-10-09T17:13:44.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=4b457b3813ec2da916f501ce132355a10e79330a 2020-10-09T17:13:44.396 INFO:teuthology.orchestra.run.smithi184:> dpkg-query -W -f '${Version}' ceph 2020-10-09T17:13:44.422 INFO:teuthology.orchestra.run.smithi184.stdout:16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:13:44.423 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-6241-g4b457b38-1bionic 2020-10-09T17:13:44.423 INFO:teuthology.task.install:The correct ceph version 16.0.0-6241-g4b457b38-1bionic is installed. 2020-10-09T17:13:44.424 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-10-09T17:13:44.424 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:44.424 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-09T17:13:44.429 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-09T17:13:44.439 INFO:teuthology.orchestra.run.smithi060.stderr:24+1 records in 2020-10-09T17:13:44.439 INFO:teuthology.orchestra.run.smithi060.stderr:24+1 records out 2020-10-09T17:13:44.439 INFO:teuthology.orchestra.run.smithi060.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.00022342 s, 56.7 MB/s 2020-10-09T17:13:44.441 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:13:44.441 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-09T17:13:44.472 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2020-10-09T17:13:44.489 INFO:teuthology.orchestra.run.smithi184.stderr:24+1 records in 2020-10-09T17:13:44.490 INFO:teuthology.orchestra.run.smithi184.stderr:24+1 records out 2020-10-09T17:13:44.491 INFO:teuthology.orchestra.run.smithi184.stderr:12667 bytes (13 kB, 12 KiB) copied, 0.000222756 s, 56.9 MB/s 2020-10-09T17:13:44.491 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-10-09T17:13:44.492 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:44.493 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2020-10-09T17:13:44.498 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-10-09T17:13:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:5+1 records in 2020-10-09T17:13:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:5+1 records out 2020-10-09T17:13:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 0.00010485 s, 29.2 MB/s 2020-10-09T17:13:44.509 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-10-09T17:13:44.567 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:13:44.568 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/daemon-helper 2020-10-09T17:13:44.573 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/usr/bin/daemon-helper 2020-10-09T17:13:44.582 INFO:teuthology.orchestra.run.smithi184.stderr:5+1 records in 2020-10-09T17:13:44.582 INFO:teuthology.orchestra.run.smithi184.stderr:5+1 records out 2020-10-09T17:13:44.582 INFO:teuthology.orchestra.run.smithi184.stderr:3058 bytes (3.1 kB, 3.0 KiB) copied, 9.9691e-05 s, 30.7 MB/s 2020-10-09T17:13:44.583 INFO:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-10-09T17:13:44.639 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-10-09T17:13:44.639 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:44.640 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2020-10-09T17:13:44.645 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-10-09T17:13:44.654 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:44.654 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:44.654 INFO:teuthology.orchestra.run.smithi060.stderr:299 bytes copied, 7.254e-05 s, 4.1 MB/s 2020-10-09T17:13:44.655 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-10-09T17:13:44.711 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:13:44.711 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/usr/bin/adjust-ulimits 2020-10-09T17:13:44.716 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/usr/bin/adjust-ulimits 2020-10-09T17:13:44.755 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:13:44.756 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:13:44.756 INFO:teuthology.orchestra.run.smithi184.stderr:299 bytes copied, 7.1116e-05 s, 4.2 MB/s 2020-10-09T17:13:44.756 INFO:teuthology.orchestra.run.smithi184:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-10-09T17:13:44.811 INFO:teuthology.run_tasks:Running task ceph... 2020-10-09T17:13:44.889 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-10-09T17:13:44.890 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2020-10-09T17:13:44.894 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 777 /var/log/ceph 2020-10-09T17:13:44.910 INFO:tasks.ceph:Disabling ceph logrotate... 2020-10-09T17:13:44.910 INFO:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2020-10-09T17:13:44.950 INFO:teuthology.orchestra.run.smithi184:> sudo rm -f -- /etc/logrotate.d/ceph 2020-10-09T17:13:44.968 INFO:tasks.ceph:Creating extra log directories... 2020-10-09T17:13:44.968 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-10-09T17:13:45.006 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-10-09T17:13:45.060 INFO:tasks.ceph:Creating ceph cluster c1... 2020-10-09T17:13:45.060 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-10-09T17:13:45.060 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504', 'branch': 'wip-investigate-datalog', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5510504', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-investigate-datalog'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-investigate-datalog', 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 85, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-investigate-datalog', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-10-09_17:00:22', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.24980', 'targets': {'smithi060.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu4kGk/54ot5aGD+tcFihNNIt3JdKbyUrwEeAO8YPODX3borROwRbqu9f4GkuXMUoe0btPn12aEp2s7+XTF5QjLdHTjwmOesvI7Zfh9ARuIxeQQ8XSYgu/qMv3A7wOFksQXvewsWtyOKssPg2Z/x+Vs4C71xtGdV269XQVtcBpQcsz9c+TDlWimzPugfuJH886Gq4ChdO4nf/6MGiCXb9/aeQhwx/wJjA7FtSWQ3Nb+tULeXr4rP3xwBfJz0DU1HhE693RY/R2k8Azq5qYlE1JWQxJLUNAG0BuQ818OAHqwtQJfTtNUYu/Pq6ww6EOSmICAP6Prx0l4/nZ2HPUoNlT', 'smithi184.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy6vRl3U+6PLXomh+38cJLLwiy+eOMCw9NaMZpGI3Ig4qmkE16SSYZ8GYnNgTRnPeY31WUo3FlRYUl0AH8kA+VQ3XhMAM6//LFjx9xf3IhC/9rfw9V4u4rTTTUkeBAxtXGC8CrtYL1q9izwV5UZgOLJU+HD8ALyeR4p34r1fsqrvOrQoGyzTkgPmH6a4BiqYbfsT7mBCZHzQTClAYjpJOeOKBqfwxlcFpRQCy/CRrEdyt2JhaOsXzwRV6KBA31OMjUzOEM8rzoGUNYFP5uJ5V42D37VKPC7mN5PRp3a2GZfOrzVkMWBbzjzOC3IAMYA0HS3F6gRutl7ESPwf1wb6u5'}} 2020-10-09T17:13:45.063 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-10-09T17:13:45.066 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-10-09T17:13:45.110 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2020-10-09T17:13:45.147 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2020-10-09T17:13:45.167 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:45.168 INFO:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2020-10-09T17:13:45.205 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-10-09T17:13:45.207 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/vg_nvme/lv_1 2020-10-09T17:13:45.207 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/vg_nvme/lv_2 2020-10-09T17:13:45.207 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/vg_nvme/lv_3 2020-10-09T17:13:45.207 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/vg_nvme/lv_4 2020-10-09T17:13:45.207 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:45.208 INFO:teuthology.orchestra.run.smithi060.stderr:72 bytes copied, 0.000181013 s, 398 kB/s 2020-10-09T17:13:45.208 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-10-09T17:13:45.208 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-10-09T17:13:45.254 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-10-09T17:13:45.255 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:13:45.255 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-10-09T17:13:45.255 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:13:45.256 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-10-09 17:11:25.448532981 +0000 2020-10-09T17:13:45.256 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-10-09 17:11:25.340532982 +0000 2020-10-09T17:13:45.256 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-10-09 17:11:25.340532982 +0000 2020-10-09T17:13:45.256 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-10-09T17:13:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-10-09T17:13:45.264 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-10-09T17:13:45.265 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00029586 s, 1.7 MB/s 2020-10-09T17:13:45.269 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-10-09T17:13:45.323 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-10-09T17:13:45.323 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:13:45.324 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-10-09T17:13:45.324 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:13:45.324 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-10-09 17:11:25.696532978 +0000 2020-10-09T17:13:45.324 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-10-09 17:11:25.588532979 +0000 2020-10-09T17:13:45.324 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-10-09 17:11:25.588532979 +0000 2020-10-09T17:13:45.325 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-10-09T17:13:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-10-09T17:13:45.332 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-10-09T17:13:45.333 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000287404 s, 1.8 MB/s 2020-10-09T17:13:45.337 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-10-09T17:13:45.386 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-10-09T17:13:45.387 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:13:45.387 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-10-09T17:13:45.387 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:13:45.387 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-10-09 17:11:25.924532976 +0000 2020-10-09T17:13:45.388 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-10-09 17:11:25.808532977 +0000 2020-10-09T17:13:45.388 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-10-09 17:11:25.808532977 +0000 2020-10-09T17:13:45.388 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-10-09T17:13:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-10-09T17:13:45.396 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-10-09T17:13:45.397 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000277245 s, 1.8 MB/s 2020-10-09T17:13:45.403 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-10-09T17:13:45.452 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-10-09T17:13:45.489 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:13:45.489 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-10-09T17:13:45.489 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:13:45.489 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-10-09 17:11:26.144532973 +0000 2020-10-09T17:13:45.490 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-10-09 17:11:26.032532975 +0000 2020-10-09T17:13:45.490 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-10-09 17:11:26.032532975 +0000 2020-10-09T17:13:45.490 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-10-09T17:13:45.491 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-10-09T17:13:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-10-09T17:13:45.492 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00028578 s, 1.8 MB/s 2020-10-09T17:13:45.492 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:13:45.492 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-09T17:13:45.493 INFO:tasks.ceph:Generating config... 2020-10-09T17:13:45.495 INFO:tasks.ceph:[client] debug rgw = 20 2020-10-09T17:13:45.495 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-10-09T17:13:45.495 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-10-09T17:13:45.496 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:13:45.496 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-10-09T17:13:45.496 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-10-09T17:13:45.496 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-10-09T17:13:45.498 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-10-09T17:13:45.498 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-10-09T17:13:45.499 INFO:tasks.ceph:[client] setgroup = ceph 2020-10-09T17:13:45.499 INFO:tasks.ceph:[client] setuser = ceph 2020-10-09T17:13:45.499 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-10-09T17:13:45.499 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-10-09T17:13:45.500 INFO:tasks.ceph:[mgr] debug ms = 1 2020-10-09T17:13:45.500 INFO:tasks.ceph:[mon] debug mon = 20 2020-10-09T17:13:45.500 INFO:tasks.ceph:[mon] debug ms = 1 2020-10-09T17:13:45.500 INFO:tasks.ceph:[mon] debug paxos = 20 2020-10-09T17:13:45.501 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-10-09T17:13:45.501 INFO:tasks.ceph:[osd] debug filestore = 20 2020-10-09T17:13:45.501 INFO:tasks.ceph:[osd] debug journal = 20 2020-10-09T17:13:45.501 INFO:tasks.ceph:[osd] debug ms = 20 2020-10-09T17:13:45.501 INFO:tasks.ceph:[osd] debug osd = 25 2020-10-09T17:13:45.504 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-10-09T17:13:45.504 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-10-09T17:13:45.504 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-10-09T17:13:45.505 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-10-09T17:13:45.505 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-10-09T17:13:45.505 INFO:tasks.ceph:Setting up c1.mon.a... 2020-10-09T17:13:45.505 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-10-09T17:13:45.564 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.keyring 2020-10-09T17:13:45.566 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-10-09T17:13:45.643 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-09T17:13:45.698 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.60')] 2020-10-09T17:13:45.699 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.60', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-10-09T17:13:45.700 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:45.700 INFO:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:13:45.744 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:13:45.746 INFO:teuthology.orchestra.run.smithi060.stderr:6+1 records in 2020-10-09T17:13:45.746 INFO:teuthology.orchestra.run.smithi060.stderr:6+1 records out 2020-10-09T17:13:45.746 INFO:teuthology.orchestra.run.smithi060.stderr:3100 bytes (3.1 kB, 3.0 KiB) copied, 0.000107263 s, 28.9 MB/s 2020-10-09T17:13:45.746 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --print /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:45.819 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:13:45.820 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:13:45.821 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 6e29134d-04aa-44b4-a744-78aeab5ae6f6 2020-10-09T17:13:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2020-10-09T17:13:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 6e29134d-04aa-44b4-a744-78aeab5ae6f6 2020-10-09T17:13:45.822 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2020-10-09T17:13:45.819861+0000 2020-10-09T17:13:45.823 INFO:teuthology.orchestra.run.smithi060.stdout:created 2020-10-09T17:13:45.819861+0000 2020-10-09T17:13:45.823 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 0 (unknown) 2020-10-09T17:13:45.823 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2020-10-09T17:13:45.828 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2020-10-09T17:13:45.828 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-10-09T17:13:45.830 INFO:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:13:45.890 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 6e29134d-04aa-44b4-a744-78aeab5ae6f6... 2020-10-09T17:13:45.894 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-10-09T17:13:45.945 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-10-09T17:13:45.977 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2020-10-09T17:13:45.978 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2020-10-09T17:13:45.978 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.979 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2020-10-09T17:13:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2020-10-09T17:13:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2020-10-09T17:13:45.980 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2020-10-09T17:13:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-09T17:13:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2020-10-09T17:13:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.981 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2020-10-09T17:13:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-09T17:13:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2020-10-09T17:13:45.982 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2020-10-09T17:13:45.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2020-10-09T17:13:45.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2020-10-09T17:13:45.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2020-10-09T17:13:45.983 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2020-10-09T17:13:45.984 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.985 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.60 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: osd heartbeat grace = 40 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 6e29134d-04aa-44b4-a744-78aeab5ae6f6 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2020-10-09T17:13:45.986 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2020-10-09T17:13:45.987 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2020-10-09T17:13:45.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2020-10-09T17:13:45.987 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.987 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.988 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.989 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2020-10-09T17:13:45.990 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 20 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 20 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 20 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 25 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: osd fast shutdown = False 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: rocksdb delete range threshold = 0 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.991 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:13:45.992 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-10-09T17:13:45.993 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd crush smoke test = False 2020-10-09T17:13:45.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-09T17:13:45.995 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw = 20 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt require ssl = False 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms backend = testing 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw curl low speed time = 300 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw data log num shards = 4 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw md log max shards = 4 2020-10-09T17:13:45.996 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync log trim interval = 0 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync obj etag verify = True 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout: setgroup = ceph 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout: setuser = ceph 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10 2020-10-09T17:13:45.997 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-10-09T17:13:45.998 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 1000 2020-10-09T17:13:45.998 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-10-09T17:13:45.998 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10000 2020-10-09T17:13:45.999 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2020-10-09T17:13:45.999 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2020-10-09T17:13:46.000 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2020-10-09T17:13:46.001 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.002 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2020-10-09T17:13:46.003 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-09T17:13:46.004 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2020-10-09T17:13:46.005 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.60 2020-10-09T17:13:46.006 INFO:teuthology.orchestra.run.smithi184.stdout: osd heartbeat grace = 40 2020-10-09T17:13:46.006 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = 6e29134d-04aa-44b4-a744-78aeab5ae6f6 2020-10-09T17:13:46.006 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.006 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2020-10-09T17:13:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2020-10-09T17:13:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2020-10-09T17:13:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: debug filestore = 20 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: debug journal = 20 2020-10-09T17:13:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 20 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 25 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: osd fast shutdown = False 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-10-09T17:13:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.012 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2020-10-09T17:13:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd crush smoke test = False 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2020-10-09T17:13:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2020-10-09T17:13:46.015 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2020-10-09T17:13:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-10-09T17:13:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2020-10-09T17:13:46.018 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-10-09T17:13:46.018 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-10-09T17:13:46.103 INFO:tasks.ceph:Copying monmap to all nodes... 2020-10-09T17:13:46.104 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:46.104 INFO:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-10-09T17:13:46.156 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd if=/etc/ceph/c1.keyring of=/dev/stdout 2020-10-09T17:13:46.157 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.] 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBJmoBfOIYxJhAAFy8TWyCGhp6HPsxg4oiVTQ== 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout:[client.admin] 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBfaLv7BRAABkB0fFBYOjm5wRKTNYY/DA== 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout: caps mds = "allow *" 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout: caps mgr = "allow *" 2020-10-09T17:13:46.158 INFO:teuthology.orchestra.run.smithi060.stdout: caps mon = "allow *" 2020-10-09T17:13:46.159 INFO:teuthology.orchestra.run.smithi060.stdout: caps osd = "allow *" 2020-10-09T17:13:46.159 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:46.159 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:46.159 INFO:teuthology.orchestra.run.smithi060.stderr:206 bytes copied, 0.000174017 s, 1.2 MB/s 2020-10-09T17:13:46.159 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:46.160 INFO:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-10-09T17:13:46.205 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2020-10-09T17:13:46.207 INFO:teuthology.orchestra.run.smithi060.stdout: �n)M�D��Dx��Z��I��_��0I��_��0�aXa �<��<a 2020-10-09T17:13:46.207 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:46.207 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:46.207 INFO:teuthology.orchestra.run.smithi060.stderr:201 bytes copied, 0.00016606 s, 1.2 MB/s 2020-10-09T17:13:46.207 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:13:46.208 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:46.208 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/c1.keyring 2020-10-09T17:13:46.208 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-09T17:13:46.257 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-10-09T17:13:46.266 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:46.267 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:46.267 INFO:teuthology.orchestra.run.smithi060.stderr:206 bytes copied, 0.00013415 s, 1.5 MB/s 2020-10-09T17:13:46.267 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-09T17:13:46.276 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:46.277 INFO:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:46.326 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:46.328 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:46.328 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:46.328 INFO:teuthology.orchestra.run.smithi060.stderr:201 bytes copied, 0.000131817 s, 1.5 MB/s 2020-10-09T17:13:46.328 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:13:46.329 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:13:46.330 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c1.keyring 2020-10-09T17:13:46.330 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-09T17:13:46.337 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/ceph/c1.keyring 2020-10-09T17:13:46.346 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:13:46.347 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:13:46.347 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 7.531e-05 s, 2.7 MB/s 2020-10-09T17:13:46.348 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo chmod 0644 /etc/ceph/c1.keyring 2020-10-09T17:13:46.359 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:13:46.359 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:46.406 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:46.407 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:13:46.407 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:13:46.407 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 7.5663e-05 s, 2.7 MB/s 2020-10-09T17:13:46.408 INFO:tasks.ceph:Setting up mon nodes... 2020-10-09T17:13:46.408 INFO:tasks.ceph:Setting up mgr nodes... 2020-10-09T17:13:46.408 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-10-09T17:13:46.452 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-10-09T17:13:46.455 INFO:tasks.ceph:Setting up mds nodes... 2020-10-09T17:13:46.455 INFO:tasks.ceph_client:Setting up client nodes... 2020-10-09T17:13:46.456 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-10-09T17:13:46.535 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.client.0.keyring 2020-10-09T17:13:46.549 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-10-09T17:13:46.627 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.client.1.keyring 2020-10-09T17:13:46.641 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-10-09T17:13:46.641 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-09T17:13:46.642 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-10-09T17:13:46.701 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:13:46.702 INFO:tasks.ceph:role: c1.osd.0 2020-10-09T17:13:46.702 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:13:46.702 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-10-09T17:13:46.786 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:13:46.787 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:13:46.788 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:13:46.788 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:13:46.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-10-09T17:13:46.789 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-10-09T17:13:46.854 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-10-09T17:13:46.909 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:13:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:13:46.910 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-10-09T17:13:46.965 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:13:46.965 INFO:tasks.ceph:role: c1.osd.1 2020-10-09T17:13:46.966 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:13:46.966 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-10-09T17:13:47.048 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:13:47.048 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:13:47.048 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:13:47.049 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:13:47.050 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-10-09T17:13:47.051 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-10-09T17:13:47.114 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-10-09T17:13:47.171 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:13:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:13:47.174 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-10-09T17:13:47.228 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:13:47.229 INFO:tasks.ceph:role: c1.osd.2 2020-10-09T17:13:47.229 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:13:47.229 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-10-09T17:13:47.316 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:13:47.316 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:13:47.316 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:13:47.316 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:13:47.317 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:13:47.317 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:13:47.317 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:13:47.317 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:13:47.317 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:13:47.318 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-10-09T17:13:47.319 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-10-09T17:13:47.380 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-10-09T17:13:47.438 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:13:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:13:47.439 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:47.529 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-10-09T17:13:47.529 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-10-09T17:13:47.529 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-09T17:13:47.530 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-09T17:13:47.530 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-10-09T17:13:47.530 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:47.526+0000 7fa49cf7ed80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-10-09T17:13:49.721 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-10-09T17:13:49.736 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-10-09T17:13:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-10-09T17:13:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-09T17:13:49.819 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-09T17:13:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-10-09T17:13:49.820 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:49.814+0000 7f8c35cb6d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-10-09T17:13:51.992 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-10-09T17:13:52.052 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:52.131 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-10-09T17:13:52.131 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-10-09T17:13:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-09T17:13:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-09T17:13:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-10-09T17:13:52.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:13:52.126+0000 7f6d93724d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-10-09T17:13:54.311 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-10-09T17:13:54.367 INFO:tasks.ceph:Reading keys from all nodes... 2020-10-09T17:13:54.368 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.368 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-10-09T17:13:54.412 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2020-10-09T17:13:54.421 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2020-10-09T17:13:54.421 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBf2ioEGxAAdnZEgii0JD5tE+RtZYYyIQ== 2020-10-09T17:13:54.422 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.422 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.422 INFO:teuthology.orchestra.run.smithi060.stderr:56 bytes copied, 0.000172316 s, 325 kB/s 2020-10-09T17:13:54.423 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.423 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-10-09T17:13:54.468 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2020-10-09T17:13:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-10-09T17:13:54.478 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBLmoBf6UiOHxAA9nkCvwDudBHIp+xkDmg9eA== 2020-10-09T17:13:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:56 bytes copied, 0.000171667 s, 326 kB/s 2020-10-09T17:13:54.479 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.479 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-10-09T17:13:54.524 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2020-10-09T17:13:54.533 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-10-09T17:13:54.533 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBNmoBfbdrUMBAAD93BpKDQwNamYBeadOEzhA== 2020-10-09T17:13:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:56 bytes copied, 0.000163513 s, 342 kB/s 2020-10-09T17:13:54.535 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.535 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-10-09T17:13:54.580 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2020-10-09T17:13:54.589 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-10-09T17:13:54.590 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBQmoBfk6XXBxAAWgLoP11joyb3MAcaDb8f3A== 2020-10-09T17:13:54.590 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.590 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.591 INFO:teuthology.orchestra.run.smithi060.stderr:56 bytes copied, 0.000164947 s, 340 kB/s 2020-10-09T17:13:54.592 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.593 INFO:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-10-09T17:13:54.640 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2020-10-09T17:13:54.641 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2020-10-09T17:13:54.642 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBfRtfzHxAACGQRTfbIIiZlzrC0/7N1Dg== 2020-10-09T17:13:54.642 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.642 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.642 INFO:teuthology.orchestra.run.smithi060.stderr:59 bytes copied, 0.000161677 s, 365 kB/s 2020-10-09T17:13:54.643 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:54.643 INFO:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-10-09T17:13:54.688 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2020-10-09T17:13:54.689 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2020-10-09T17:13:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBfvvdyJRAA4g1VLEfG0mmKMM98OHPeHg== 2020-10-09T17:13:54.690 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:54.690 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:54.691 INFO:teuthology.orchestra.run.smithi060.stderr:59 bytes copied, 0.000167084 s, 353 kB/s 2020-10-09T17:13:54.691 INFO:tasks.ceph:Adding keys to all mons... 2020-10-09T17:13:54.692 INFO:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/c1.keyring 2020-10-09T17:13:54.745 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBf2ioEGxAAdnZEgii0JD5tE+RtZYYyIQ== 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBLmoBf6UiOHxAA9nkCvwDudBHIp+xkDmg9eA== 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBNmoBfbdrUMBAAD93BpKDQwNamYBeadOEzhA== 2020-10-09T17:13:54.746 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-10-09T17:13:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBQmoBfk6XXBxAAWgLoP11joyb3MAcaDb8f3A== 2020-10-09T17:13:54.747 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2020-10-09T17:13:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBfRtfzHxAACGQRTfbIIiZlzrC0/7N1Dg== 2020-10-09T17:13:54.747 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2020-10-09T17:13:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBKmoBfvvdyJRAA4g1VLEfG0mmKMM98OHPeHg== 2020-10-09T17:13:54.748 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-10-09T17:13:54.834 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:13:54.920 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:13:55.004 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:13:55.086 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-09T17:13:55.173 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-09T17:13:55.216 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-10-09T17:13:55.216 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-10-09T17:13:55.273 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-10-09T17:13:55.373 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-10-09T17:13:55.387 INFO:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-10-09T17:13:55.434 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-10-09T17:13:55.435 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-10-09T17:13:55.436 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-10-09T17:13:55.478 INFO:tasks.ceph.c1.mon.a:Started 2020-10-09T17:13:55.478 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-10-09T17:13:55.478 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-10-09T17:13:55.479 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-10-09T17:13:55.481 INFO:tasks.ceph.c1.mgr.x:Started 2020-10-09T17:13:55.481 INFO:tasks.ceph:Setting crush tunables to default 2020-10-09T17:13:55.482 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd crush tunables default 2020-10-09T17:13:55.908 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2020-10-09T17:13:55.920 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-10-09T17:13:55.920 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:55.921 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-10-09T17:13:55.924 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2020-10-09T17:13:55.929 INFO:teuthology.orchestra.run.smithi060.stdout:4b08f2bd-7950-440d-bb87-ea76eeecf73a 2020-10-09T17:13:55.930 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:55.930 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:55.930 INFO:teuthology.orchestra.run.smithi060.stderr:37 bytes copied, 0.000203417 s, 182 kB/s 2020-10-09T17:13:55.931 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:55.931 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-10-09T17:13:55.976 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2020-10-09T17:13:55.985 INFO:teuthology.orchestra.run.smithi060.stdout:be1d6bbf-9b2d-42be-b828-48e415a967b8 2020-10-09T17:13:55.986 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:55.986 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:55.986 INFO:teuthology.orchestra.run.smithi060.stderr:37 bytes copied, 0.000209767 s, 176 kB/s 2020-10-09T17:13:55.987 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:13:55.987 INFO:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-10-09T17:13:56.032 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2020-10-09T17:13:56.039 INFO:teuthology.orchestra.run.smithi060.stdout:f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f 2020-10-09T17:13:56.040 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:13:56.040 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:13:56.040 INFO:teuthology.orchestra.run.smithi060.stderr:37 bytes copied, 0.000212417 s, 174 kB/s 2020-10-09T17:13:56.041 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new 4b08f2bd-7950-440d-bb87-ea76eeecf73a 0 2020-10-09T17:13:56.300 INFO:teuthology.orchestra.run.smithi060.stdout:0 2020-10-09T17:13:56.313 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new be1d6bbf-9b2d-42be-b828-48e415a967b8 1 2020-10-09T17:13:56.526 INFO:teuthology.orchestra.run.smithi060.stdout:1 2020-10-09T17:13:56.540 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f 2 2020-10-09T17:13:56.761 INFO:teuthology.orchestra.run.smithi060.stdout:2 2020-10-09T17:13:56.778 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-10-09T17:13:56.779 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-10-09T17:13:56.781 INFO:tasks.ceph.c1.osd.0:Started 2020-10-09T17:13:56.782 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-10-09T17:13:56.782 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-10-09T17:13:56.784 INFO:tasks.ceph.c1.osd.1:Started 2020-10-09T17:13:56.785 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-10-09T17:13:56.785 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-10-09T17:13:56.787 INFO:tasks.ceph.c1.osd.2:Started 2020-10-09T17:13:56.788 INFO:tasks.ceph:Waiting for OSDs to come up 2020-10-09T17:13:57.091 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-10-09T17:13:57.182 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-10-09T17:13:57.178+0000 7fb3ad774d80 -1 Falling back to public interface 2020-10-09T17:13:57.269 INFO:teuthology.misc.health.smithi060.stdout: 2020-10-09T17:13:57.270 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":5,"fsid":"6e29134d-04aa-44b4-a744-78aeab5ae6f6","created":"2020-10-09T17:13:55.612049+0000","modified":"2020-10-09T17:13:56.758509+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"4b08f2bd-7950-440d-bb87-ea76eeecf73a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be1d6bbf-9b2d-42be-b828-48e415a967b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:13:57.281 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-10-09T17:13:57.402 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-10-09T17:13:57.398+0000 7fca1a277d80 -1 Falling back to public interface 2020-10-09T17:13:57.418 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-10-09T17:13:57.414+0000 7f6122276d80 -1 Falling back to public interface 2020-10-09T17:13:57.809 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-10-09T17:13:57.806+0000 7fb3ad774d80 -1 osd.0 0 log_to_monitors {default=true} 2020-10-09T17:13:58.000 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-10-09T17:13:57.994+0000 7fca1a277d80 -1 osd.2 0 log_to_monitors {default=true} 2020-10-09T17:13:58.045 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-10-09T17:13:58.042+0000 7f6122276d80 -1 osd.1 0 log_to_monitors {default=true} 2020-10-09T17:13:59.637 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-10-09T17:13:59.634+0000 7fb3a6442700 -1 osd.0 0 waiting for initial osdmap 2020-10-09T17:13:59.637 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-10-09T17:13:59.634+0000 7fca11742700 -1 osd.2 0 waiting for initial osdmap 2020-10-09T17:13:59.637 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-10-09T17:13:59.634+0000 7f6119741700 -1 osd.1 0 waiting for initial osdmap 2020-10-09T17:14:03.585 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-10-09T17:14:03.876 INFO:teuthology.misc.health.smithi060.stdout: 2020-10-09T17:14:03.877 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"6e29134d-04aa-44b4-a744-78aeab5ae6f6","created":"2020-10-09T17:13:55.612049+0000","modified":"2020-10-09T17:14:02.644203+0000","last_up_change":"2020-10-09T17:14:00.633770+0000","last_in_change":"2020-10-09T17:14:00.633770+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:13:59.597291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"4b08f2bd-7950-440d-bb87-ea76eeecf73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":14578},{"type":"v1","addr":"172.21.15.60:6801","nonce":14578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":14578},{"type":"v1","addr":"172.21.15.60:6803","nonce":14578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":14578},{"type":"v1","addr":"172.21.15.60:6807","nonce":14578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":14578},{"type":"v1","addr":"172.21.15.60:6805","nonce":14578}]},"public_addr":"172.21.15.60:6801/14578","cluster_addr":"172.21.15.60:6803/14578","heartbeat_back_addr":"172.21.15.60:6807/14578","heartbeat_front_addr":"172.21.15.60:6805/14578","state":["exists","up"]},{"osd":1,"uuid":"be1d6bbf-9b2d-42be-b828-48e415a967b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":14579},{"type":"v1","addr":"172.21.15.60:6817","nonce":14579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":14579},{"type":"v1","addr":"172.21.15.60:6819","nonce":14579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":14579},{"type":"v1","addr":"172.21.15.60:6823","nonce":14579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":14579},{"type":"v1","addr":"172.21.15.60:6821","nonce":14579}]},"public_addr":"172.21.15.60:6817/14579","cluster_addr":"172.21.15.60:6819/14579","heartbeat_back_addr":"172.21.15.60:6823/14579","heartbeat_front_addr":"172.21.15.60:6821/14579","state":["exists","up"]},{"osd":2,"uuid":"f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":14580},{"type":"v1","addr":"172.21.15.60:6809","nonce":14580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":14580},{"type":"v1","addr":"172.21.15.60:6811","nonce":14580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":14580},{"type":"v1","addr":"172.21.15.60:6815","nonce":14580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":14580},{"type":"v1","addr":"172.21.15.60:6813","nonce":14580}]},"public_addr":"172.21.15.60:6809/14580","cluster_addr":"172.21.15.60:6811/14580","heartbeat_back_addr":"172.21.15.60:6815/14580","heartbeat_front_addr":"172.21.15.60:6813/14580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:58.824729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.070537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.021192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:03.890 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-10-09T17:14:03.891 INFO:tasks.ceph:Creating RBD pool 2020-10-09T17:14:03.891 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd pool create rbd 8 2020-10-09T17:14:04.676 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2020-10-09T17:14:04.690 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-10-09T17:14:05.682 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rbd' on pool 'rbd' 2020-10-09T17:14:05.697 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-10-09T17:14:05.698 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-09T17:14:05.699 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-10-09T17:14:05.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:05.981 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"fsid":"6e29134d-04aa-44b4-a744-78aeab5ae6f6","created":"2020-10-09T17:13:55.612049+0000","modified":"2020-10-09T17:14:05.663411+0000","last_up_change":"2020-10-09T17:14:00.633770+0000","last_in_change":"2020-10-09T17:14:00.633770+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:13:59.597291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:04.184693+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b08f2bd-7950-440d-bb87-ea76eeecf73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":14578},{"type":"v1","addr":"172.21.15.60:6801","nonce":14578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":14578},{"type":"v1","addr":"172.21.15.60:6803","nonce":14578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":14578},{"type":"v1","addr":"172.21.15.60:6807","nonce":14578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":14578},{"type":"v1","addr":"172.21.15.60:6805","nonce":14578}]},"public_addr":"172.21.15.60:6801/14578","cluster_addr":"172.21.15.60:6803/14578","heartbeat_back_addr":"172.21.15.60:6807/14578","heartbeat_front_addr":"172.21.15.60:6805/14578","state":["exists","up"]},{"osd":1,"uuid":"be1d6bbf-9b2d-42be-b828-48e415a967b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":14579},{"type":"v1","addr":"172.21.15.60:6817","nonce":14579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":14579},{"type":"v1","addr":"172.21.15.60:6819","nonce":14579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":14579},{"type":"v1","addr":"172.21.15.60:6823","nonce":14579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":14579},{"type":"v1","addr":"172.21.15.60:6821","nonce":14579}]},"public_addr":"172.21.15.60:6817/14579","cluster_addr":"172.21.15.60:6819/14579","heartbeat_back_addr":"172.21.15.60:6823/14579","heartbeat_front_addr":"172.21.15.60:6821/14579","state":["exists","up"]},{"osd":2,"uuid":"f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":14580},{"type":"v1","addr":"172.21.15.60:6809","nonce":14580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":14580},{"type":"v1","addr":"172.21.15.60:6811","nonce":14580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":14580},{"type":"v1","addr":"172.21.15.60:6815","nonce":14580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":14580},{"type":"v1","addr":"172.21.15.60:6813","nonce":14580}]},"public_addr":"172.21.15.60:6809/14580","cluster_addr":"172.21.15.60:6811/14580","heartbeat_back_addr":"172.21.15.60:6815/14580","heartbeat_front_addr":"172.21.15.60:6813/14580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:58.824729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.070537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.021192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:05.996 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-10-09T17:13:59.597291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-10-09T17:14:04.184693+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-10-09T17:14:05.996 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-10-09T17:14:06.290 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2020-10-09T17:14:06.304 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-10-09T17:14:06.600 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2020-10-09T17:14:06.614 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-10-09T17:14:06.615 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-10-09T17:14:06.615 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-10-09T17:14:06.938 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:06.952 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":14396},{"type":"v1","addr":"172.21.15.60:6825","nonce":14396}]},"active_addr":"172.21.15.60:6825/14396","active_change":"2020-10-09T17:13:59.574845+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1715462978}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2533062554}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2800482001}]}]}} 2020-10-09T17:14:06.954 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-10-09T17:14:06.955 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-10-09T17:14:06.955 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-09T17:14:07.252 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:07.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"6e29134d-04aa-44b4-a744-78aeab5ae6f6","created":"2020-10-09T17:13:55.612049+0000","modified":"2020-10-09T17:14:06.668705+0000","last_up_change":"2020-10-09T17:14:00.633770+0000","last_in_change":"2020-10-09T17:14:00.633770+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:13:59.597291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:04.184693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b08f2bd-7950-440d-bb87-ea76eeecf73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":14578},{"type":"v1","addr":"172.21.15.60:6801","nonce":14578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":14578},{"type":"v1","addr":"172.21.15.60:6803","nonce":14578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":14578},{"type":"v1","addr":"172.21.15.60:6807","nonce":14578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":14578},{"type":"v1","addr":"172.21.15.60:6805","nonce":14578}]},"public_addr":"172.21.15.60:6801/14578","cluster_addr":"172.21.15.60:6803/14578","heartbeat_back_addr":"172.21.15.60:6807/14578","heartbeat_front_addr":"172.21.15.60:6805/14578","state":["exists","up"]},{"osd":1,"uuid":"be1d6bbf-9b2d-42be-b828-48e415a967b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":14579},{"type":"v1","addr":"172.21.15.60:6817","nonce":14579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":14579},{"type":"v1","addr":"172.21.15.60:6819","nonce":14579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":14579},{"type":"v1","addr":"172.21.15.60:6823","nonce":14579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":14579},{"type":"v1","addr":"172.21.15.60:6821","nonce":14579}]},"public_addr":"172.21.15.60:6817/14579","cluster_addr":"172.21.15.60:6819/14579","heartbeat_back_addr":"172.21.15.60:6823/14579","heartbeat_front_addr":"172.21.15.60:6821/14579","state":["exists","up"]},{"osd":2,"uuid":"f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":14580},{"type":"v1","addr":"172.21.15.60:6809","nonce":14580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":14580},{"type":"v1","addr":"172.21.15.60:6811","nonce":14580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":14580},{"type":"v1","addr":"172.21.15.60:6815","nonce":14580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":14580},{"type":"v1","addr":"172.21.15.60:6813","nonce":14580}]},"public_addr":"172.21.15.60:6809/14580","cluster_addr":"172.21.15.60:6811/14580","heartbeat_back_addr":"172.21.15.60:6815/14580","heartbeat_front_addr":"172.21.15.60:6813/14580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:58.824729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.070537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.021192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:07.266 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-10-09T17:14:07.266 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-10-09T17:14:07.559 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:07.559 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"6e29134d-04aa-44b4-a744-78aeab5ae6f6","created":"2020-10-09T17:13:55.612049+0000","modified":"2020-10-09T17:14:06.668705+0000","last_up_change":"2020-10-09T17:14:00.633770+0000","last_in_change":"2020-10-09T17:14:00.633770+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:13:59.597291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:04.184693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b08f2bd-7950-440d-bb87-ea76eeecf73a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":14578},{"type":"v1","addr":"172.21.15.60:6801","nonce":14578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":14578},{"type":"v1","addr":"172.21.15.60:6803","nonce":14578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":14578},{"type":"v1","addr":"172.21.15.60:6807","nonce":14578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":14578},{"type":"v1","addr":"172.21.15.60:6805","nonce":14578}]},"public_addr":"172.21.15.60:6801/14578","cluster_addr":"172.21.15.60:6803/14578","heartbeat_back_addr":"172.21.15.60:6807/14578","heartbeat_front_addr":"172.21.15.60:6805/14578","state":["exists","up"]},{"osd":1,"uuid":"be1d6bbf-9b2d-42be-b828-48e415a967b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":14579},{"type":"v1","addr":"172.21.15.60:6817","nonce":14579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":14579},{"type":"v1","addr":"172.21.15.60:6819","nonce":14579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":14579},{"type":"v1","addr":"172.21.15.60:6823","nonce":14579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":14579},{"type":"v1","addr":"172.21.15.60:6821","nonce":14579}]},"public_addr":"172.21.15.60:6817/14579","cluster_addr":"172.21.15.60:6819/14579","heartbeat_back_addr":"172.21.15.60:6823/14579","heartbeat_front_addr":"172.21.15.60:6821/14579","state":["exists","up"]},{"osd":2,"uuid":"f5c6fafd-8b07-4b0e-b02b-8e81ce34dc1f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":14580},{"type":"v1","addr":"172.21.15.60:6809","nonce":14580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":14580},{"type":"v1","addr":"172.21.15.60:6811","nonce":14580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":14580},{"type":"v1","addr":"172.21.15.60:6815","nonce":14580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":14580},{"type":"v1","addr":"172.21.15.60:6813","nonce":14580}]},"public_addr":"172.21.15.60:6809/14580","cluster_addr":"172.21.15.60:6811/14580","heartbeat_back_addr":"172.21.15.60:6815/14580","heartbeat_front_addr":"172.21.15.60:6813/14580","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:58.824729+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.070537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:13:59.021192+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:07.574 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-10-09T17:14:07.751 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-10-09T17:14:07.751 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-10-09T17:14:07.932 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-10-09T17:14:07.933 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-10-09T17:14:08.115 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-10-09T17:14:08.116 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-10-09T17:14:08.425 INFO:teuthology.orchestra.run.smithi060.stdout:34359738370 2020-10-09T17:14:08.439 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2020-10-09T17:14:09.440 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-10-09T17:14:09.740 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-10-09T17:14:09.755 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2020-10-09T17:14:09.755 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-10-09T17:14:10.047 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-10-09T17:14:10.062 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-10-09T17:14:10.062 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-10-09T17:14:10.349 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-10-09T17:14:10.362 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-10-09T17:14:10.363 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-10-09T17:14:10.363 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-10-09T17:14:10.650 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:10.653 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2020-10-09T17:14:10.665 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-10-09T17:14:09.585294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.910583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686614+0000","last_change":"2020-10-09T17:14:05.685534+0000","last_active":"2020-10-09T17:14:05.686614+0000","last_peered":"2020-10-09T17:14:05.686614+0000","last_clean":"2020-10-09T17:14:05.686614+0000","last_became_active":"2020-10-09T17:14:05.685032+0000","last_became_peered":"2020-10-09T17:14:05.685032+0000","last_unstale":"2020-10-09T17:14:05.686614+0000","last_undegraded":"2020-10-09T17:14:05.686614+0000","last_fullsized":"2020-10-09T17:14:05.686614+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686396+0000","last_change":"2020-10-09T17:14:05.685500+0000","last_active":"2020-10-09T17:14:05.686396+0000","last_peered":"2020-10-09T17:14:05.686396+0000","last_clean":"2020-10-09T17:14:05.686396+0000","last_became_active":"2020-10-09T17:14:05.684498+0000","last_became_peered":"2020-10-09T17:14:05.684498+0000","last_unstale":"2020-10-09T17:14:05.686396+0000","last_undegraded":"2020-10-09T17:14:05.686396+0000","last_fullsized":"2020-10-09T17:14:05.686396+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.687410+0000","last_change":"2020-10-09T17:14:05.686798+0000","last_active":"2020-10-09T17:14:05.687410+0000","last_peered":"2020-10-09T17:14:05.687410+0000","last_clean":"2020-10-09T17:14:05.687410+0000","last_became_active":"2020-10-09T17:14:05.686414+0000","last_became_peered":"2020-10-09T17:14:05.686414+0000","last_unstale":"2020-10-09T17:14:05.687410+0000","last_undegraded":"2020-10-09T17:14:05.687410+0000","last_fullsized":"2020-10-09T17:14:05.687410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.683071+0000","last_change":"2020-10-09T17:14:05.680344+0000","last_active":"2020-10-09T17:14:05.683071+0000","last_peered":"2020-10-09T17:14:05.683071+0000","last_clean":"2020-10-09T17:14:05.683071+0000","last_became_active":"2020-10-09T17:14:05.679358+0000","last_became_peered":"2020-10-09T17:14:05.679358+0000","last_unstale":"2020-10-09T17:14:05.683071+0000","last_undegraded":"2020-10-09T17:14:05.683071+0000","last_fullsized":"2020-10-09T17:14:05.683071+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.679892+0000","last_change":"2020-10-09T17:14:01.794224+0000","last_active":"2020-10-09T17:14:05.679892+0000","last_peered":"2020-10-09T17:14:05.679892+0000","last_clean":"2020-10-09T17:14:05.679892+0000","last_became_active":"2020-10-09T17:14:01.792406+0000","last_became_peered":"2020-10-09T17:14:01.792406+0000","last_unstale":"2020-10-09T17:14:05.679892+0000","last_undegraded":"2020-10-09T17:14:05.679892+0000","last_fullsized":"2020-10-09T17:14:05.679892+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:13:59.627620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:13:59.627620+0000","last_clean_scrub_stamp":"2020-10-09T17:13:59.627620+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.685134+0000","last_change":"2020-10-09T17:14:05.682750+0000","last_active":"2020-10-09T17:14:05.685134+0000","last_peered":"2020-10-09T17:14:05.685134+0000","last_clean":"2020-10-09T17:14:05.685134+0000","last_became_active":"2020-10-09T17:14:05.682311+0000","last_became_peered":"2020-10-09T17:14:05.682311+0000","last_unstale":"2020-10-09T17:14:05.685134+0000","last_undegraded":"2020-10-09T17:14:05.685134+0000","last_fullsized":"2020-10-09T17:14:05.685134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.683899+0000","last_change":"2020-10-09T17:14:05.682469+0000","last_active":"2020-10-09T17:14:05.683899+0000","last_peered":"2020-10-09T17:14:05.683899+0000","last_clean":"2020-10-09T17:14:05.683899+0000","last_became_active":"2020-10-09T17:14:05.682049+0000","last_became_peered":"2020-10-09T17:14:05.682049+0000","last_unstale":"2020-10-09T17:14:05.683899+0000","last_undegraded":"2020-10-09T17:14:05.683899+0000","last_fullsized":"2020-10-09T17:14:05.683899+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.685826+0000","last_change":"2020-10-09T17:14:05.684769+0000","last_active":"2020-10-09T17:14:05.685826+0000","last_peered":"2020-10-09T17:14:05.685826+0000","last_clean":"2020-10-09T17:14:05.685826+0000","last_became_active":"2020-10-09T17:14:05.683964+0000","last_became_peered":"2020-10-09T17:14:05.683964+0000","last_unstale":"2020-10-09T17:14:05.685826+0000","last_undegraded":"2020-10-09T17:14:05.685826+0000","last_fullsized":"2020-10-09T17:14:05.685826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686323+0000","last_change":"2020-10-09T17:14:05.685353+0000","last_active":"2020-10-09T17:14:05.686323+0000","last_peered":"2020-10-09T17:14:05.686323+0000","last_clean":"2020-10-09T17:14:05.686323+0000","last_became_active":"2020-10-09T17:14:05.683295+0000","last_became_peered":"2020-10-09T17:14:05.683295+0000","last_unstale":"2020-10-09T17:14:05.686323+0000","last_undegraded":"2020-10-09T17:14:05.686323+0000","last_fullsized":"2020-10-09T17:14:05.686323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]}]}} 2020-10-09T17:14:10.666 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-10-09T17:14:10.958 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:10.960 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2020-10-09T17:14:10.971 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-10-09T17:14:09.585294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14752,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":314558048,"statfs":{"total":322122547200,"available":322107441152,"internally_reserved":0,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.910583"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686614+0000","last_change":"2020-10-09T17:14:05.685534+0000","last_active":"2020-10-09T17:14:05.686614+0000","last_peered":"2020-10-09T17:14:05.686614+0000","last_clean":"2020-10-09T17:14:05.686614+0000","last_became_active":"2020-10-09T17:14:05.685032+0000","last_became_peered":"2020-10-09T17:14:05.685032+0000","last_unstale":"2020-10-09T17:14:05.686614+0000","last_undegraded":"2020-10-09T17:14:05.686614+0000","last_fullsized":"2020-10-09T17:14:05.686614+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686396+0000","last_change":"2020-10-09T17:14:05.685500+0000","last_active":"2020-10-09T17:14:05.686396+0000","last_peered":"2020-10-09T17:14:05.686396+0000","last_clean":"2020-10-09T17:14:05.686396+0000","last_became_active":"2020-10-09T17:14:05.684498+0000","last_became_peered":"2020-10-09T17:14:05.684498+0000","last_unstale":"2020-10-09T17:14:05.686396+0000","last_undegraded":"2020-10-09T17:14:05.686396+0000","last_fullsized":"2020-10-09T17:14:05.686396+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.687410+0000","last_change":"2020-10-09T17:14:05.686798+0000","last_active":"2020-10-09T17:14:05.687410+0000","last_peered":"2020-10-09T17:14:05.687410+0000","last_clean":"2020-10-09T17:14:05.687410+0000","last_became_active":"2020-10-09T17:14:05.686414+0000","last_became_peered":"2020-10-09T17:14:05.686414+0000","last_unstale":"2020-10-09T17:14:05.687410+0000","last_undegraded":"2020-10-09T17:14:05.687410+0000","last_fullsized":"2020-10-09T17:14:05.687410+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.683071+0000","last_change":"2020-10-09T17:14:05.680344+0000","last_active":"2020-10-09T17:14:05.683071+0000","last_peered":"2020-10-09T17:14:05.683071+0000","last_clean":"2020-10-09T17:14:05.683071+0000","last_became_active":"2020-10-09T17:14:05.679358+0000","last_became_peered":"2020-10-09T17:14:05.679358+0000","last_unstale":"2020-10-09T17:14:05.683071+0000","last_undegraded":"2020-10-09T17:14:05.683071+0000","last_fullsized":"2020-10-09T17:14:05.683071+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.679892+0000","last_change":"2020-10-09T17:14:01.794224+0000","last_active":"2020-10-09T17:14:05.679892+0000","last_peered":"2020-10-09T17:14:05.679892+0000","last_clean":"2020-10-09T17:14:05.679892+0000","last_became_active":"2020-10-09T17:14:01.792406+0000","last_became_peered":"2020-10-09T17:14:01.792406+0000","last_unstale":"2020-10-09T17:14:05.679892+0000","last_undegraded":"2020-10-09T17:14:05.679892+0000","last_fullsized":"2020-10-09T17:14:05.679892+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:13:59.627620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:13:59.627620+0000","last_clean_scrub_stamp":"2020-10-09T17:13:59.627620+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.685134+0000","last_change":"2020-10-09T17:14:05.682750+0000","last_active":"2020-10-09T17:14:05.685134+0000","last_peered":"2020-10-09T17:14:05.685134+0000","last_clean":"2020-10-09T17:14:05.685134+0000","last_became_active":"2020-10-09T17:14:05.682311+0000","last_became_peered":"2020-10-09T17:14:05.682311+0000","last_unstale":"2020-10-09T17:14:05.685134+0000","last_undegraded":"2020-10-09T17:14:05.685134+0000","last_fullsized":"2020-10-09T17:14:05.685134+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.683899+0000","last_change":"2020-10-09T17:14:05.682469+0000","last_active":"2020-10-09T17:14:05.683899+0000","last_peered":"2020-10-09T17:14:05.683899+0000","last_clean":"2020-10-09T17:14:05.683899+0000","last_became_active":"2020-10-09T17:14:05.682049+0000","last_became_peered":"2020-10-09T17:14:05.682049+0000","last_unstale":"2020-10-09T17:14:05.683899+0000","last_undegraded":"2020-10-09T17:14:05.683899+0000","last_fullsized":"2020-10-09T17:14:05.683899+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.685826+0000","last_change":"2020-10-09T17:14:05.684769+0000","last_active":"2020-10-09T17:14:05.685826+0000","last_peered":"2020-10-09T17:14:05.685826+0000","last_clean":"2020-10-09T17:14:05.685826+0000","last_became_active":"2020-10-09T17:14:05.683964+0000","last_became_peered":"2020-10-09T17:14:05.683964+0000","last_unstale":"2020-10-09T17:14:05.685826+0000","last_undegraded":"2020-10-09T17:14:05.685826+0000","last_fullsized":"2020-10-09T17:14:05.685826+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:05.686323+0000","last_change":"2020-10-09T17:14:05.685353+0000","last_active":"2020-10-09T17:14:05.686323+0000","last_peered":"2020-10-09T17:14:05.686323+0000","last_clean":"2020-10-09T17:14:05.686323+0000","last_became_active":"2020-10-09T17:14:05.683295+0000","last_became_peered":"2020-10-09T17:14:05.683295+0000","last_unstale":"2020-10-09T17:14:05.686323+0000","last_undegraded":"2020-10-09T17:14:05.686323+0000","last_fullsized":"2020-10-09T17:14:05.686323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:04.656867+0000","last_clean_scrub_stamp":"2020-10-09T17:14:04.656867+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]}]}]}} 2020-10-09T17:14:10.973 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-10-09T17:14:10.973 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-10-09T17:14:10.973 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-10-09T17:14:10.973 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-10-09T17:14:11.311 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:11.326 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-10-09T17:14:11.326 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-10-09T17:14:11.327 INFO:teuthology.run_tasks:Running task ceph... 2020-10-09T17:14:11.338 INFO:tasks.ceph:Creating ceph cluster c2... 2020-10-09T17:14:11.338 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-10-09T17:14:11.338 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi/5510504', 'branch': 'wip-investigate-datalog', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'aemerson@redhat.com', 'first_in_suite': False, 'job_id': '5510504', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'aemerson-2020-10-09_17:00:22-rgw-wip-investigate-datalog-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-investigate-datalog'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-investigate-datalog', 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a'}}, 'owner': 'scheduled_aemerson@teuthology', 'priority': 85, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-investigate-datalog', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '4b457b3813ec2da916f501ce132355a10e79330a', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2020-10-09_17:00:22', 'tube': 'smithi', 'user': 'aemerson', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.24980', 'targets': {'smithi060.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu4kGk/54ot5aGD+tcFihNNIt3JdKbyUrwEeAO8YPODX3borROwRbqu9f4GkuXMUoe0btPn12aEp2s7+XTF5QjLdHTjwmOesvI7Zfh9ARuIxeQQ8XSYgu/qMv3A7wOFksQXvewsWtyOKssPg2Z/x+Vs4C71xtGdV269XQVtcBpQcsz9c+TDlWimzPugfuJH886Gq4ChdO4nf/6MGiCXb9/aeQhwx/wJjA7FtSWQ3Nb+tULeXr4rP3xwBfJz0DU1HhE693RY/R2k8Azq5qYlE1JWQxJLUNAG0BuQ818OAHqwtQJfTtNUYu/Pq6ww6EOSmICAP6Prx0l4/nZ2HPUoNlT', 'smithi184.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCy6vRl3U+6PLXomh+38cJLLwiy+eOMCw9NaMZpGI3Ig4qmkE16SSYZ8GYnNgTRnPeY31WUo3FlRYUl0AH8kA+VQ3XhMAM6//LFjx9xf3IhC/9rfw9V4u4rTTTUkeBAxtXGC8CrtYL1q9izwV5UZgOLJU+HD8ALyeR4p34r1fsqrvOrQoGyzTkgPmH6a4BiqYbfsT7mBCZHzQTClAYjpJOeOKBqfwxlcFpRQCy/CRrEdyt2JhaOsXzwRV6KBA31OMjUzOEM8rzoGUNYFP5uJ5V42D37VKPC7mN5PRp3a2GZfOrzVkMWBbzjzOC3IAMYA0HS3F6gRutl7ESPwf1wb6u5'}} 2020-10-09T17:14:11.339 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-10-09T17:14:11.341 INFO:teuthology.orchestra.run.smithi184:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-10-09T17:14:11.348 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2020-10-09T17:14:11.389 INFO:teuthology.orchestra.run.smithi184:> sudo install -d -m0777 -- /var/run/ceph 2020-10-09T17:14:11.407 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:11.407 INFO:teuthology.orchestra.run.smithi184:> dd if=/scratch_devs of=/dev/stdout 2020-10-09T17:14:11.452 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/scratch_devs of=/dev/stdout 2020-10-09T17:14:11.453 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_1 2020-10-09T17:14:11.453 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_2 2020-10-09T17:14:11.454 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_3 2020-10-09T17:14:11.454 INFO:teuthology.orchestra.run.smithi184.stdout:/dev/vg_nvme/lv_4 2020-10-09T17:14:11.454 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:11.454 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:11.454 INFO:teuthology.orchestra.run.smithi184.stderr:72 bytes copied, 0.00018104 s, 398 kB/s 2020-10-09T17:14:11.455 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-10-09T17:14:11.455 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-10-09T17:14:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-10-09T17:14:11.502 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-10-09 17:11:26.416147329 +0000 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-10-09 17:11:26.308147330 +0000 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-10-09 17:11:26.308147330 +0000 2020-10-09T17:14:11.503 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-10-09T17:14:11.512 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-10-09T17:14:11.512 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-10-09T17:14:11.512 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000317031 s, 1.6 MB/s 2020-10-09T17:14:11.516 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-10-09T17:14:11.562 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-10-09 17:11:26.544147327 +0000 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-10-09 17:11:26.540147327 +0000 2020-10-09T17:14:11.563 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-10-09 17:11:26.540147327 +0000 2020-10-09T17:14:11.564 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-10-09T17:14:11.572 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-10-09T17:14:11.572 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-10-09T17:14:11.572 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.000353104 s, 1.4 MB/s 2020-10-09T17:14:11.576 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-10-09T17:14:11.630 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-10-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-10-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-10-09 17:11:26.792147325 +0000 2020-10-09T17:14:11.631 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-10-09 17:11:26.788147325 +0000 2020-10-09T17:14:11.632 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-10-09 17:11:26.788147325 +0000 2020-10-09T17:14:11.632 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-10-09T17:14:11.639 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-10-09T17:14:11.640 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-10-09T17:14:11.640 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00038767 s, 1.3 MB/s 2020-10-09T17:14:11.643 INFO:teuthology.orchestra.run.smithi184:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-10-09T17:14:11.690 INFO:teuthology.orchestra.run.smithi184.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout:Access: 2020-10-09 17:11:27.148147321 +0000 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout:Modify: 2020-10-09 17:11:27.044147322 +0000 2020-10-09T17:14:11.691 INFO:teuthology.orchestra.run.smithi184.stdout:Change: 2020-10-09 17:11:27.044147322 +0000 2020-10-09T17:14:11.692 INFO:teuthology.orchestra.run.smithi184.stdout: Birth: - 2020-10-09T17:14:11.700 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records in 2020-10-09T17:14:11.700 INFO:teuthology.orchestra.run.smithi184.stderr:1+0 records out 2020-10-09T17:14:11.700 INFO:teuthology.orchestra.run.smithi184.stderr:512 bytes copied, 0.00042811 s, 1.2 MB/s 2020-10-09T17:14:11.703 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:14:11.703 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-09T17:14:11.704 INFO:tasks.ceph:Generating config... 2020-10-09T17:14:11.705 INFO:tasks.ceph:[client] debug rgw = 20 2020-10-09T17:14:11.705 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-10-09T17:14:11.705 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-10-09T17:14:11.705 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:14:11.706 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-10-09T17:14:11.706 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-10-09T17:14:11.706 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-10-09T17:14:11.706 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-10-09T17:14:11.706 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2020-10-09T17:14:11.707 INFO:tasks.ceph:[client] setgroup = ceph 2020-10-09T17:14:11.707 INFO:tasks.ceph:[client] setuser = ceph 2020-10-09T17:14:11.707 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-10-09T17:14:11.707 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-10-09T17:14:11.707 INFO:tasks.ceph:[mgr] debug ms = 1 2020-10-09T17:14:11.708 INFO:tasks.ceph:[mon] debug mon = 20 2020-10-09T17:14:11.708 INFO:tasks.ceph:[mon] debug ms = 1 2020-10-09T17:14:11.708 INFO:tasks.ceph:[mon] debug paxos = 20 2020-10-09T17:14:11.708 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-10-09T17:14:11.708 INFO:tasks.ceph:[osd] debug filestore = 20 2020-10-09T17:14:11.709 INFO:tasks.ceph:[osd] debug journal = 20 2020-10-09T17:14:11.709 INFO:tasks.ceph:[osd] debug ms = 20 2020-10-09T17:14:11.709 INFO:tasks.ceph:[osd] debug osd = 25 2020-10-09T17:14:11.709 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-10-09T17:14:11.709 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-10-09T17:14:11.710 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-10-09T17:14:11.710 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-10-09T17:14:11.710 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-10-09T17:14:11.710 INFO:tasks.ceph:Setting up c2.mon.a... 2020-10-09T17:14:11.711 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-10-09T17:14:11.777 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.keyring 2020-10-09T17:14:11.779 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-10-09T17:14:11.854 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-09T17:14:11.905 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.184')] 2020-10-09T17:14:11.906 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon host': '172.21.15.184', 'osd heartbeat grace': 40}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2020-10-09T17:14:11.907 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:11.907 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:14:11.952 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:14:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:6+1 records in 2020-10-09T17:14:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:6+1 records out 2020-10-09T17:14:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:3101 bytes (3.1 kB, 3.0 KiB) copied, 0.00010763 s, 28.8 MB/s 2020-10-09T17:14:11.955 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.184 --print /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:14:12.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:14:12.026 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.026 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: generated fsid d00779bf-ec3b-4550-b0de-5068ea21a118 2020-10-09T17:14:12.026 INFO:teuthology.orchestra.run.smithi184.stdout:epoch 0 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:fsid d00779bf-ec3b-4550-b0de-5068ea21a118 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:last_changed 2020-10-09T17:14:12.028201+0000 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:created 2020-10-09T17:14:12.028201+0000 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:min_mon_release 0 (unknown) 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:election_strategy: 1 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:0: [v2:172.21.15.184:3300/0,v1:172.21.15.184:6789/0] mon.a 2020-10-09T17:14:12.027 INFO:teuthology.orchestra.run.smithi184.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-10-09T17:14:12.029 INFO:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2020-10-09T17:14:12.078 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID d00779bf-ec3b-4550-b0de-5068ea21a118... 2020-10-09T17:14:12.079 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-10-09T17:14:12.081 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-10-09T17:14:12.122 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2020-10-09T17:14:12.123 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2020-10-09T17:14:12.124 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-09T17:14:12.125 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2020-10-09T17:14:12.126 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.127 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2020-10-09T17:14:12.128 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.184 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: osd heartbeat grace = 40 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = d00779bf-ec3b-4550-b0de-5068ea21a118 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.129 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2020-10-09T17:14:12.130 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.131 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2020-10-09T17:14:12.132 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 20 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 20 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 20 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 25 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: osd fast shutdown = False 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: rocksdb delete range threshold = 0 2020-10-09T17:14:12.133 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:14:12.134 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-10-09T17:14:12.135 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd crush smoke test = False 2020-10-09T17:14:12.136 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw = 20 2020-10-09T17:14:12.137 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt require ssl = False 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms backend = testing 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw curl low speed time = 300 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw data log num shards = 4 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw md log max shards = 4 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync log trim interval = 0 2020-10-09T17:14:12.138 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync obj etag verify = True 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout: setgroup = ceph 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout: setuser = ceph 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-10-09T17:14:12.139 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 1000 2020-10-09T17:14:12.140 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-10-09T17:14:12.140 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10000 2020-10-09T17:14:12.152 INFO:teuthology.orchestra.run.smithi184.stdout:[global] 2020-10-09T17:14:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: chdir = "" 2020-10-09T17:14:12.152 INFO:teuthology.orchestra.run.smithi184.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: auth supported = cephx 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: filestore xattr use omap = true 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: mon clock drift allowed = 1.000 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.153 INFO:teuthology.orchestra.run.smithi184.stdout: osd crush chooseleaf type = 0 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: auth debug = true 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on old message = true 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: ms die on bug = true 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: mon max pg per osd = 10000 # >= luminous 2020-10-09T17:14:12.154 INFO:teuthology.orchestra.run.smithi184.stdout: mon pg warn max object skew = 0 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: # disable pg_autoscaler by default for new pools 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: osd_pool_default_pg_autoscale_mode = off 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default size = 2 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.155 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow primary affinity = true 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd allow pg remap = true 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on legacy crush tunables = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on crush straw calc version zero = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on no sortbitwise = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on osd down out interval zero = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon warn on too few osds = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-10-09T17:14:12.156 INFO:teuthology.orchestra.run.smithi184.stdout: mon_warn_on_pool_no_redundancy = false 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: mon_allow_pool_size_one = true 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: osd default data pool replay window = 5 2020-10-09T17:14:12.157 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: mon allow pool delete = true 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: mon cluster log file level = debug 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: debug asserts on shutdown = true 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: mon host = 172.21.15.184 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: osd heartbeat grace = 40 2020-10-09T17:14:12.158 INFO:teuthology.orchestra.run.smithi184.stdout: fsid = d00779bf-ec3b-4550-b0de-5068ea21a118 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout:[osd] 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: osd journal size = 100 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub load threshold = 5.0 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: osd scrub max interval = 600 2020-10-09T17:14:12.159 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd recover clone overlap = true 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd recovery max chunk = 1048576 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug shutdown = true 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug op order = true 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug verify stray on activate = true 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.160 INFO:teuthology.orchestra.run.smithi184.stdout: osd open classes on start = true 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug pg log writeout = true 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: osd deep scrub update digest min age = 30 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: osd map max advance = 10 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.161 INFO:teuthology.orchestra.run.smithi184.stdout: journal zero on create = true 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: filestore ondisk finisher threads = 3 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: filestore apply finisher threads = 3 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: bdev debug aio = true 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: osd debug misdirected ops = true 2020-10-09T17:14:12.162 INFO:teuthology.orchestra.run.smithi184.stdout: debug filestore = 20 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: debug journal = 20 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 20 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: debug osd = 25 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: osd fast shutdown = False 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: rocksdb delete range threshold = 0 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr] 2020-10-09T17:14:12.163 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: debug mgr = 20 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: mgr/telemetry/nag = false 2020-10-09T17:14:12.164 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout:[mon] 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: debug ms = 1 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: debug mon = 20 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: debug paxos = 20 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: debug auth = 20 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: mon data avail warn = 5 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: mon mgr mkfs grace = 240 2020-10-09T17:14:12.165 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min pgs per osd = 4 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd reporter subtree level = osd 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd prime pg temp = true 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: mon reweight min bytes per osd = 10 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: mon osd crush smoke test = False 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout:[client] 2020-10-09T17:14:12.166 INFO:teuthology.orchestra.run.smithi184.stdout: rgw cache enabled = true 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable ops log = true 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: rgw enable usage log = true 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: debug rgw = 20 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt require ssl = False 2020-10-09T17:14:12.167 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms backend = testing 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw curl low speed time = 300 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw data log num shards = 4 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw md log max shards = 4 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync log trim interval = 0 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: rgw sync obj etag verify = True 2020-10-09T17:14:12.168 INFO:teuthology.orchestra.run.smithi184.stdout: setgroup = ceph 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: setuser = ceph 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.a] 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 1000 2020-10-09T17:14:12.169 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-10-09T17:14:12.170 INFO:teuthology.orchestra.run.smithi184.stdout: osd_max_omap_entries_per_request = 10000 2020-10-09T17:14:12.170 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-10-09T17:14:12.171 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-10-09T17:14:12.257 INFO:tasks.ceph:Copying monmap to all nodes... 2020-10-09T17:14:12.258 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:12.258 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-10-09T17:14:12.263 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.keyring of=/dev/stdout 2020-10-09T17:14:12.265 INFO:teuthology.orchestra.run.smithi184.stdout:[mon.] 2020-10-09T17:14:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBjmoBfOlrlMhAAmbT0F90qZ6oIncXphjFL3Q== 2020-10-09T17:14:12.265 INFO:teuthology.orchestra.run.smithi184.stdout:[client.admin] 2020-10-09T17:14:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBfJjFaDxAAhNo+37cTCbN+0ZQ88IMKVA== 2020-10-09T17:14:12.265 INFO:teuthology.orchestra.run.smithi184.stdout: caps mds = "allow *" 2020-10-09T17:14:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: caps mgr = "allow *" 2020-10-09T17:14:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: caps mon = "allow *" 2020-10-09T17:14:12.266 INFO:teuthology.orchestra.run.smithi184.stdout: caps osd = "allow *" 2020-10-09T17:14:12.266 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:12.267 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:12.267 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 0.000216299 s, 952 kB/s 2020-10-09T17:14:12.267 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:12.267 INFO:teuthology.orchestra.run.smithi184:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-10-09T17:14:12.312 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2020-10-09T17:14:12.313 INFO:teuthology.orchestra.run.smithi184.stdout: ��y��;EP��Ph�!�d��_�P�d��_�P��aXa �����a 2020-10-09T17:14:12.314 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:12.314 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:12.314 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 0.000171264 s, 1.2 MB/s 2020-10-09T17:14:12.314 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2020-10-09T17:14:12.315 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:14:12.315 INFO:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/c2.keyring 2020-10-09T17:14:12.315 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-09T17:14:12.320 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-10-09T17:14:12.329 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:14:12.329 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:14:12.329 INFO:teuthology.orchestra.run.smithi060.stderr:206 bytes copied, 7.2273e-05 s, 2.9 MB/s 2020-10-09T17:14:12.330 INFO:teuthology.orchestra.run.smithi060.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-09T17:14:12.339 INFO:teuthology.orchestra.run.smithi060:> set -ex 2020-10-09T17:14:12.340 INFO:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.384 INFO:teuthology.orchestra.run.smithi060.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.385 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2020-10-09T17:14:12.385 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2020-10-09T17:14:12.386 INFO:teuthology.orchestra.run.smithi060.stderr:201 bytes copied, 7.2717e-05 s, 2.8 MB/s 2020-10-09T17:14:12.386 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:14:12.386 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:12.386 INFO:teuthology.orchestra.run.smithi184:> sudo dd of=/etc/ceph/c2.keyring 2020-10-09T17:14:12.387 INFO:teuthology.orchestra.run.smithi184:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-09T17:14:12.391 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd of=/etc/ceph/c2.keyring 2020-10-09T17:14:12.400 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:12.401 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:12.401 INFO:teuthology.orchestra.run.smithi184.stderr:206 bytes copied, 0.000135404 s, 1.5 MB/s 2020-10-09T17:14:12.402 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo chmod 0644 /etc/ceph/c2.keyring 2020-10-09T17:14:12.412 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:12.412 INFO:teuthology.orchestra.run.smithi184:> dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.456 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd of=/home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:12.458 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:12.458 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:12.458 INFO:teuthology.orchestra.run.smithi184.stderr:201 bytes copied, 0.000134104 s, 1.5 MB/s 2020-10-09T17:14:12.458 INFO:tasks.ceph:Setting up mon nodes... 2020-10-09T17:14:12.459 INFO:tasks.ceph:Setting up mgr nodes... 2020-10-09T17:14:12.459 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-10-09T17:14:12.543 INFO:teuthology.orchestra.run.smithi184.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-10-09T17:14:12.545 INFO:tasks.ceph:Setting up mds nodes... 2020-10-09T17:14:12.545 INFO:tasks.ceph_client:Setting up client nodes... 2020-10-09T17:14:12.546 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-10-09T17:14:12.620 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.0.keyring 2020-10-09T17:14:12.632 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-10-09T17:14:12.708 INFO:teuthology.orchestra.run.smithi184.stdout:creating /etc/ceph/c2.client.1.keyring 2020-10-09T17:14:12.720 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-10-09T17:14:12.720 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi184.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-10-09T17:14:12.720 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-10-09T17:14:12.775 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:14:12.775 INFO:tasks.ceph:role: c2.osd.0 2020-10-09T17:14:12.775 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:14:12.776 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-10-09T17:14:13.300 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:14:13.300 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:14:13.300 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:14:13.300 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:14:13.300 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:14:13.301 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:14:13.301 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:14:13.301 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:14:13.301 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:14:13.302 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-10-09T17:14:13.302 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-10-09T17:14:13.383 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-10-09T17:14:13.398 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:14:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:14:13.399 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-10-09T17:14:13.454 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:14:13.455 INFO:tasks.ceph:role: c2.osd.1 2020-10-09T17:14:13.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:14:13.455 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-10-09T17:14:14.008 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:14:14.008 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:14:14.009 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:14:14.010 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-10-09T17:14:14.011 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-10-09T17:14:14.047 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-10-09T17:14:14.098 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:14:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:14:14.099 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-10-09T17:14:14.155 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-10-09T17:14:14.155 INFO:tasks.ceph:role: c2.osd.2 2020-10-09T17:14:14.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com 2020-10-09T17:14:14.156 INFO:teuthology.orchestra.run.smithi184:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-10-09T17:14:14.683 INFO:teuthology.orchestra.run.smithi184.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-10-09T17:14:14.683 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 attr=2, projid32bit=1 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: = sunit=0 swidth=0 blks 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-10-09T17:14:14.684 INFO:teuthology.orchestra.run.smithi184.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-10-09T17:14:14.685 INFO:teuthology.orchestra.run.smithi184.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-10-09T17:14:14.685 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi184.front.sepia.ceph.com -o noatime 2020-10-09T17:14:14.686 INFO:teuthology.orchestra.run.smithi184:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-10-09T17:14:14.726 INFO:teuthology.orchestra.run.smithi184:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-10-09T17:14:14.779 INFO:teuthology.orchestra.run.smithi184.stderr:sudo: /sbin/restorecon: command not found 2020-10-09T17:14:14.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-10-09T17:14:14.780 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:14.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.870+0000 7fb196cf5d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-10-09T17:14:14.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.870+0000 7fb196cf5d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-10-09T17:14:14.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.870+0000 7fb196cf5d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-09T17:14:14.872 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.870+0000 7fb196cf5d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-09T17:14:14.873 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.870+0000 7fb196cf5d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-10-09T17:14:14.881 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:14.882+0000 7fb196cf5d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-10-09T17:14:17.056 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-10-09T17:14:17.072 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:17.154 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-10-09T17:14:17.154 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-10-09T17:14:17.154 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-09T17:14:17.155 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-09T17:14:17.155 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-10-09T17:14:17.155 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:17.154+0000 7f9c2e9c0d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-10-09T17:14:19.340 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-10-09T17:14:19.399 INFO:teuthology.orchestra.run.smithi184:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:19.478 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-10-09T17:14:19.479 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-10-09T17:14:19.479 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-09T17:14:19.479 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-09T17:14:19.479 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-10-09T17:14:19.480 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:14:19.478+0000 7f49aefbed80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-10-09T17:14:21.643 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-10-09T17:14:21.699 INFO:tasks.ceph:Reading keys from all nodes... 2020-10-09T17:14:21.699 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.699 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-10-09T17:14:21.744 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2020-10-09T17:14:21.751 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2020-10-09T17:14:21.751 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBffmKHIBAANKlXKVxiBoucXWW2ihlhng== 2020-10-09T17:14:21.751 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:21.751 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:21.752 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.0001738 s, 322 kB/s 2020-10-09T17:14:21.752 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.753 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-10-09T17:14:21.802 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2020-10-09T17:14:21.809 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-10-09T17:14:21.809 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBmmoBfQLoWNBAAjUEEVr1iYSFRV56Uuj+4aQ== 2020-10-09T17:14:21.809 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:21.809 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:21.810 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.000165577 s, 338 kB/s 2020-10-09T17:14:21.811 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.811 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-10-09T17:14:21.856 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2020-10-09T17:14:21.864 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-10-09T17:14:21.865 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBpmoBfkH5SCRAAyIfDcqbsxBaAeYFYjdd1AQ== 2020-10-09T17:14:21.865 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:21.865 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:21.865 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.00016672 s, 336 kB/s 2020-10-09T17:14:21.866 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.866 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-10-09T17:14:21.916 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2020-10-09T17:14:21.925 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-10-09T17:14:21.925 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBrmoBfD76gHBAAyRYffrYPnElBdIUItpWBng== 2020-10-09T17:14:21.925 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:21.926 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:21.926 INFO:teuthology.orchestra.run.smithi184.stderr:56 bytes copied, 0.00016248 s, 345 kB/s 2020-10-09T17:14:21.927 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.927 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-10-09T17:14:21.972 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2020-10-09T17:14:21.973 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2020-10-09T17:14:21.973 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBfdBcjJRAAX5+bLaAOORsfRIdT7Ap0sQ== 2020-10-09T17:14:21.974 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:21.974 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:21.974 INFO:teuthology.orchestra.run.smithi184.stderr:59 bytes copied, 0.0001704 s, 346 kB/s 2020-10-09T17:14:21.974 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:21.975 INFO:teuthology.orchestra.run.smithi184:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-10-09T17:14:22.020 INFO:teuthology.orchestra.run.smithi184.stderr:+ dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2020-10-09T17:14:22.021 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2020-10-09T17:14:22.021 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBfEgFgKhAA0I/7C+FW/+75zcoaFy1MeA== 2020-10-09T17:14:22.022 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:22.022 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:22.022 INFO:teuthology.orchestra.run.smithi184.stderr:59 bytes copied, 0.000177167 s, 333 kB/s 2020-10-09T17:14:22.022 INFO:tasks.ceph:Adding keys to all mons... 2020-10-09T17:14:22.023 INFO:teuthology.orchestra.run.smithi184:> sudo tee -a /etc/ceph/c2.keyring 2020-10-09T17:14:22.077 INFO:teuthology.orchestra.run.smithi184.stdout:[mgr.x] 2020-10-09T17:14:22.077 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBffmKHIBAANKlXKVxiBoucXWW2ihlhng== 2020-10-09T17:14:22.077 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.0] 2020-10-09T17:14:22.077 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBmmoBfQLoWNBAAjUEEVr1iYSFRV56Uuj+4aQ== 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.1] 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBpmoBfkH5SCRAAyIfDcqbsxBaAeYFYjdd1AQ== 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout:[osd.2] 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBrmoBfD76gHBAAyRYffrYPnElBdIUItpWBng== 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout:[client.0] 2020-10-09T17:14:22.078 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBfdBcjJRAAX5+bLaAOORsfRIdT7Ap0sQ== 2020-10-09T17:14:22.079 INFO:teuthology.orchestra.run.smithi184.stdout:[client.1] 2020-10-09T17:14:22.079 INFO:teuthology.orchestra.run.smithi184.stdout: key = AQBkmoBfEgFgKhAA0I/7C+FW/+75zcoaFy1MeA== 2020-10-09T17:14:22.080 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-10-09T17:14:22.165 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:14:22.249 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:14:22.332 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-10-09T17:14:22.417 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-09T17:14:22.504 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-10-09T17:14:22.592 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-10-09T17:14:22.592 INFO:teuthology.orchestra.run.smithi184:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-10-09T17:14:22.651 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-10-09T17:14:22.745 INFO:teuthology.orchestra.run.smithi184:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-10-09T17:14:22.758 INFO:teuthology.orchestra.run.smithi184:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-10-09T17:14:22.806 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-10-09T17:14:22.806 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-10-09T17:14:22.807 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-10-09T17:14:22.849 INFO:tasks.ceph.c2.mon.a:Started 2020-10-09T17:14:22.850 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-10-09T17:14:22.850 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-10-09T17:14:22.851 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-10-09T17:14:22.853 INFO:tasks.ceph.c2.mgr.x:Started 2020-10-09T17:14:22.853 INFO:tasks.ceph:Setting crush tunables to default 2020-10-09T17:14:22.853 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd crush tunables default 2020-10-09T17:14:23.137 INFO:teuthology.orchestra.run.smithi184.stderr:adjusted tunables profile to default 2020-10-09T17:14:23.148 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-10-09T17:14:23.148 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:23.149 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-10-09T17:14:23.153 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2020-10-09T17:14:23.161 INFO:teuthology.orchestra.run.smithi184.stdout:9776734a-44e9-4e0c-8cbb-3a09c9e00df4 2020-10-09T17:14:23.162 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:23.162 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:23.162 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 0.00010009 s, 370 kB/s 2020-10-09T17:14:23.163 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:23.163 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-10-09T17:14:23.207 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2020-10-09T17:14:23.215 INFO:teuthology.orchestra.run.smithi184.stdout:67cc4aab-fb3a-4bfc-aec1-cd404849eec9 2020-10-09T17:14:23.215 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:23.215 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:23.215 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 0.000169297 s, 219 kB/s 2020-10-09T17:14:23.216 INFO:teuthology.orchestra.run.smithi184:> set -ex 2020-10-09T17:14:23.217 INFO:teuthology.orchestra.run.smithi184:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-10-09T17:14:23.263 INFO:teuthology.orchestra.run.smithi184.stderr:+ sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2020-10-09T17:14:23.271 INFO:teuthology.orchestra.run.smithi184.stdout:651e9a39-4df6-4358-a592-98030ba9f75e 2020-10-09T17:14:23.271 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records in 2020-10-09T17:14:23.272 INFO:teuthology.orchestra.run.smithi184.stderr:0+1 records out 2020-10-09T17:14:23.272 INFO:teuthology.orchestra.run.smithi184.stderr:37 bytes copied, 0.00017387 s, 213 kB/s 2020-10-09T17:14:23.273 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 9776734a-44e9-4e0c-8cbb-3a09c9e00df4 0 2020-10-09T17:14:23.534 INFO:teuthology.orchestra.run.smithi184.stdout:0 2020-10-09T17:14:23.546 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 67cc4aab-fb3a-4bfc-aec1-cd404849eec9 1 2020-10-09T17:14:23.759 INFO:teuthology.orchestra.run.smithi184.stdout:1 2020-10-09T17:14:23.772 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd new 651e9a39-4df6-4358-a592-98030ba9f75e 2 2020-10-09T17:14:23.986 INFO:teuthology.orchestra.run.smithi184.stdout:2 2020-10-09T17:14:23.999 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-10-09T17:14:23.999 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-10-09T17:14:24.001 INFO:tasks.ceph.c2.osd.0:Started 2020-10-09T17:14:24.001 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-10-09T17:14:24.002 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-10-09T17:14:24.003 INFO:tasks.ceph.c2.osd.1:Started 2020-10-09T17:14:24.006 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-10-09T17:14:24.007 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-10-09T17:14:24.009 INFO:tasks.ceph.c2.osd.2:Started 2020-10-09T17:14:24.009 INFO:tasks.ceph:Waiting for OSDs to come up 2020-10-09T17:14:24.312 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-10-09T17:14:24.374 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-10-09T17:14:24.374+0000 7f440ec67d80 -1 Falling back to public interface 2020-10-09T17:14:24.509 INFO:teuthology.misc.health.smithi184.stdout: 2020-10-09T17:14:24.510 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":5,"fsid":"d00779bf-ec3b-4550-b0de-5068ea21a118","created":"2020-10-09T17:14:22.969369+0000","modified":"2020-10-09T17:14:23.986185+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"9776734a-44e9-4e0c-8cbb-3a09c9e00df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"67cc4aab-fb3a-4bfc-aec1-cd404849eec9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"651e9a39-4df6-4358-a592-98030ba9f75e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:24.521 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-10-09T17:14:24.610 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-10-09T17:14:24.610+0000 7f6bc2079d80 -1 Falling back to public interface 2020-10-09T17:14:24.622 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-10-09T17:14:24.622+0000 7fd38f38dd80 -1 Falling back to public interface 2020-10-09T17:14:25.019 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-10-09T17:14:25.018+0000 7f440ec67d80 -1 osd.0 0 log_to_monitors {default=true} 2020-10-09T17:14:25.231 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-10-09T17:14:25.230+0000 7f6bc2079d80 -1 osd.1 0 log_to_monitors {default=true} 2020-10-09T17:14:25.274 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-10-09T17:14:25.274+0000 7fd38f38dd80 -1 osd.2 0 log_to_monitors {default=true} 2020-10-09T17:14:26.997 INFO:tasks.ceph.c2.osd.1.smithi184.stderr:2020-10-09T17:14:26.998+0000 7f6bb9544700 -1 osd.1 0 waiting for initial osdmap 2020-10-09T17:14:26.998 INFO:tasks.ceph.c2.osd.2.smithi184.stderr:2020-10-09T17:14:26.998+0000 7fd386858700 -1 osd.2 0 waiting for initial osdmap 2020-10-09T17:14:26.998 INFO:tasks.ceph.c2.osd.0.smithi184.stderr:2020-10-09T17:14:26.998+0000 7f4406132700 -1 osd.0 0 waiting for initial osdmap 2020-10-09T17:14:30.825 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-10-09T17:14:31.116 INFO:teuthology.misc.health.smithi184.stdout: 2020-10-09T17:14:31.117 INFO:teuthology.misc.health.smithi184.stdout:{"epoch":10,"fsid":"d00779bf-ec3b-4550-b0de-5068ea21a118","created":"2020-10-09T17:14:22.969369+0000","modified":"2020-10-09T17:14:30.659701+0000","last_up_change":"2020-10-09T17:14:27.993332+0000","last_in_change":"2020-10-09T17:14:27.993332+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:14:26.684088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"9776734a-44e9-4e0c-8cbb-3a09c9e00df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":14175},{"type":"v1","addr":"172.21.15.184:6801","nonce":14175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":14175},{"type":"v1","addr":"172.21.15.184:6803","nonce":14175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":14175},{"type":"v1","addr":"172.21.15.184:6807","nonce":14175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":14175},{"type":"v1","addr":"172.21.15.184:6805","nonce":14175}]},"public_addr":"172.21.15.184:6801/14175","cluster_addr":"172.21.15.184:6803/14175","heartbeat_back_addr":"172.21.15.184:6807/14175","heartbeat_front_addr":"172.21.15.184:6805/14175","state":["exists","up"]},{"osd":1,"uuid":"67cc4aab-fb3a-4bfc-aec1-cd404849eec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":14174},{"type":"v1","addr":"172.21.15.184:6809","nonce":14174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":14174},{"type":"v1","addr":"172.21.15.184:6811","nonce":14174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":14174},{"type":"v1","addr":"172.21.15.184:6815","nonce":14174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":14174},{"type":"v1","addr":"172.21.15.184:6813","nonce":14174}]},"public_addr":"172.21.15.184:6809/14174","cluster_addr":"172.21.15.184:6811/14174","heartbeat_back_addr":"172.21.15.184:6815/14174","heartbeat_front_addr":"172.21.15.184:6813/14174","state":["exists","up"]},{"osd":2,"uuid":"651e9a39-4df6-4358-a592-98030ba9f75e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":14176},{"type":"v1","addr":"172.21.15.184:6817","nonce":14176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":14176},{"type":"v1","addr":"172.21.15.184:6819","nonce":14176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":14176},{"type":"v1","addr":"172.21.15.184:6823","nonce":14176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":14176},{"type":"v1","addr":"172.21.15.184:6821","nonce":14176}]},"public_addr":"172.21.15.184:6817/14176","cluster_addr":"172.21.15.184:6819/14176","heartbeat_back_addr":"172.21.15.184:6823/14176","heartbeat_front_addr":"172.21.15.184:6821/14176","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.067285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.246140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.287786+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:31.130 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-10-09T17:14:31.130 INFO:tasks.ceph:Creating RBD pool 2020-10-09T17:14:31.131 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool create rbd 8 2020-10-09T17:14:32.152 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'rbd' created 2020-10-09T17:14:32.168 INFO:teuthology.orchestra.run.smithi184:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-10-09T17:14:33.163 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rbd' on pool 'rbd' 2020-10-09T17:14:33.176 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-10-09T17:14:33.177 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-09T17:14:33.178 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-10-09T17:14:33.456 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:33.457 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":12,"fsid":"d00779bf-ec3b-4550-b0de-5068ea21a118","created":"2020-10-09T17:14:22.969369+0000","modified":"2020-10-09T17:14:33.145316+0000","last_up_change":"2020-10-09T17:14:27.993332+0000","last_in_change":"2020-10-09T17:14:27.993332+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:14:26.684088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:31.411312+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9776734a-44e9-4e0c-8cbb-3a09c9e00df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":14175},{"type":"v1","addr":"172.21.15.184:6801","nonce":14175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":14175},{"type":"v1","addr":"172.21.15.184:6803","nonce":14175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":14175},{"type":"v1","addr":"172.21.15.184:6807","nonce":14175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":14175},{"type":"v1","addr":"172.21.15.184:6805","nonce":14175}]},"public_addr":"172.21.15.184:6801/14175","cluster_addr":"172.21.15.184:6803/14175","heartbeat_back_addr":"172.21.15.184:6807/14175","heartbeat_front_addr":"172.21.15.184:6805/14175","state":["exists","up"]},{"osd":1,"uuid":"67cc4aab-fb3a-4bfc-aec1-cd404849eec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":14174},{"type":"v1","addr":"172.21.15.184:6809","nonce":14174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":14174},{"type":"v1","addr":"172.21.15.184:6811","nonce":14174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":14174},{"type":"v1","addr":"172.21.15.184:6815","nonce":14174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":14174},{"type":"v1","addr":"172.21.15.184:6813","nonce":14174}]},"public_addr":"172.21.15.184:6809/14174","cluster_addr":"172.21.15.184:6811/14174","heartbeat_back_addr":"172.21.15.184:6815/14174","heartbeat_front_addr":"172.21.15.184:6813/14174","state":["exists","up"]},{"osd":2,"uuid":"651e9a39-4df6-4358-a592-98030ba9f75e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":14176},{"type":"v1","addr":"172.21.15.184:6817","nonce":14176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":14176},{"type":"v1","addr":"172.21.15.184:6819","nonce":14176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":14176},{"type":"v1","addr":"172.21.15.184:6823","nonce":14176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":14176},{"type":"v1","addr":"172.21.15.184:6821","nonce":14176}]},"public_addr":"172.21.15.184:6817/14176","cluster_addr":"172.21.15.184:6819/14176","heartbeat_back_addr":"172.21.15.184:6823/14176","heartbeat_front_addr":"172.21.15.184:6821/14176","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.067285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.246140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.287786+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:33.471 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-10-09T17:14:26.684088+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-10-09T17:14:31.411312+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-10-09T17:14:33.472 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-10-09T17:14:33.760 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 1 2020-10-09T17:14:33.773 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-10-09T17:14:34.063 INFO:teuthology.orchestra.run.smithi184.stdout:pg_num: 8 2020-10-09T17:14:34.077 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-10-09T17:14:34.078 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-10-09T17:14:34.078 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-10-09T17:14:34.403 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:34.417 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6824","nonce":13991},{"type":"v1","addr":"172.21.15.184:6825","nonce":13991}]},"active_addr":"172.21.15.184:6825/13991","active_change":"2020-10-09T17:14:26.640725+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":1862206459}]},{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":72683402}]},{"addrvec":[{"type":"v2","addr":"172.21.15.184:0","nonce":3094290927}]}]}} 2020-10-09T17:14:34.419 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-10-09T17:14:34.419 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-10-09T17:14:34.420 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-09T17:14:34.704 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:34.704 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":13,"fsid":"d00779bf-ec3b-4550-b0de-5068ea21a118","created":"2020-10-09T17:14:22.969369+0000","modified":"2020-10-09T17:14:34.151145+0000","last_up_change":"2020-10-09T17:14:27.993332+0000","last_in_change":"2020-10-09T17:14:27.993332+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:14:26.684088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:31.411312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9776734a-44e9-4e0c-8cbb-3a09c9e00df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":14175},{"type":"v1","addr":"172.21.15.184:6801","nonce":14175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":14175},{"type":"v1","addr":"172.21.15.184:6803","nonce":14175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":14175},{"type":"v1","addr":"172.21.15.184:6807","nonce":14175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":14175},{"type":"v1","addr":"172.21.15.184:6805","nonce":14175}]},"public_addr":"172.21.15.184:6801/14175","cluster_addr":"172.21.15.184:6803/14175","heartbeat_back_addr":"172.21.15.184:6807/14175","heartbeat_front_addr":"172.21.15.184:6805/14175","state":["exists","up"]},{"osd":1,"uuid":"67cc4aab-fb3a-4bfc-aec1-cd404849eec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":14174},{"type":"v1","addr":"172.21.15.184:6809","nonce":14174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":14174},{"type":"v1","addr":"172.21.15.184:6811","nonce":14174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":14174},{"type":"v1","addr":"172.21.15.184:6815","nonce":14174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":14174},{"type":"v1","addr":"172.21.15.184:6813","nonce":14174}]},"public_addr":"172.21.15.184:6809/14174","cluster_addr":"172.21.15.184:6811/14174","heartbeat_back_addr":"172.21.15.184:6815/14174","heartbeat_front_addr":"172.21.15.184:6813/14174","state":["exists","up"]},{"osd":2,"uuid":"651e9a39-4df6-4358-a592-98030ba9f75e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":14176},{"type":"v1","addr":"172.21.15.184:6817","nonce":14176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":14176},{"type":"v1","addr":"172.21.15.184:6819","nonce":14176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":14176},{"type":"v1","addr":"172.21.15.184:6823","nonce":14176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":14176},{"type":"v1","addr":"172.21.15.184:6821","nonce":14176}]},"public_addr":"172.21.15.184:6817/14176","cluster_addr":"172.21.15.184:6819/14176","heartbeat_back_addr":"172.21.15.184:6823/14176","heartbeat_front_addr":"172.21.15.184:6821/14176","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.067285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.246140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.287786+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:34.717 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-10-09T17:14:34.718 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-10-09T17:14:34.995 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:34.996 INFO:teuthology.orchestra.run.smithi184.stdout:{"epoch":13,"fsid":"d00779bf-ec3b-4550-b0de-5068ea21a118","created":"2020-10-09T17:14:22.969369+0000","modified":"2020-10-09T17:14:34.151145+0000","last_up_change":"2020-10-09T17:14:27.993332+0000","last_in_change":"2020-10-09T17:14:27.993332+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-10-09T17:14:26.684088+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-10-09T17:14:31.411312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9776734a-44e9-4e0c-8cbb-3a09c9e00df4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6800","nonce":14175},{"type":"v1","addr":"172.21.15.184:6801","nonce":14175}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6802","nonce":14175},{"type":"v1","addr":"172.21.15.184:6803","nonce":14175}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6806","nonce":14175},{"type":"v1","addr":"172.21.15.184:6807","nonce":14175}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6804","nonce":14175},{"type":"v1","addr":"172.21.15.184:6805","nonce":14175}]},"public_addr":"172.21.15.184:6801/14175","cluster_addr":"172.21.15.184:6803/14175","heartbeat_back_addr":"172.21.15.184:6807/14175","heartbeat_front_addr":"172.21.15.184:6805/14175","state":["exists","up"]},{"osd":1,"uuid":"67cc4aab-fb3a-4bfc-aec1-cd404849eec9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6808","nonce":14174},{"type":"v1","addr":"172.21.15.184:6809","nonce":14174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6810","nonce":14174},{"type":"v1","addr":"172.21.15.184:6811","nonce":14174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6814","nonce":14174},{"type":"v1","addr":"172.21.15.184:6815","nonce":14174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6812","nonce":14174},{"type":"v1","addr":"172.21.15.184:6813","nonce":14174}]},"public_addr":"172.21.15.184:6809/14174","cluster_addr":"172.21.15.184:6811/14174","heartbeat_back_addr":"172.21.15.184:6815/14174","heartbeat_front_addr":"172.21.15.184:6813/14174","state":["exists","up"]},{"osd":2,"uuid":"651e9a39-4df6-4358-a592-98030ba9f75e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6816","nonce":14176},{"type":"v1","addr":"172.21.15.184:6817","nonce":14176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6818","nonce":14176},{"type":"v1","addr":"172.21.15.184:6819","nonce":14176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6822","nonce":14176},{"type":"v1","addr":"172.21.15.184:6823","nonce":14176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.184:6820","nonce":14176},{"type":"v1","addr":"172.21.15.184:6821","nonce":14176}]},"public_addr":"172.21.15.184:6817/14176","cluster_addr":"172.21.15.184:6819/14176","heartbeat_back_addr":"172.21.15.184:6823/14176","heartbeat_front_addr":"172.21.15.184:6821/14176","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.067285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.246140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2020-10-09T17:14:26.287786+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2020-10-09T17:14:35.009 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-10-09T17:14:35.171 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-10-09T17:14:35.172 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-10-09T17:14:35.340 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-10-09T17:14:35.340 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-10-09T17:14:35.504 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-10-09T17:14:35.504 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-09T17:14:35.796 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2020-10-09T17:14:35.810 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-10-09T17:14:36.811 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-09T17:14:37.105 INFO:teuthology.orchestra.run.smithi184.stdout:34359738370 2020-10-09T17:14:37.119 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-10-09T17:14:38.120 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-10-09T17:14:38.423 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-10-09T17:14:38.437 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-10-09T17:14:38.437 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-10-09T17:14:38.717 INFO:teuthology.orchestra.run.smithi184.stdout:34359738372 2020-10-09T17:14:38.731 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2020-10-09T17:14:38.731 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-10-09T17:14:39.019 INFO:teuthology.orchestra.run.smithi184.stdout:34359738371 2020-10-09T17:14:39.033 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2020-10-09T17:14:39.034 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-10-09T17:14:39.034 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-10-09T17:14:39.314 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:39.316 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2020-10-09T17:14:39.327 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-10-09T17:14:38.651773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.494978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167112+0000","last_change":"2020-10-09T17:14:33.165329+0000","last_active":"2020-10-09T17:14:33.167112+0000","last_peered":"2020-10-09T17:14:33.167112+0000","last_clean":"2020-10-09T17:14:33.167112+0000","last_became_active":"2020-10-09T17:14:33.164513+0000","last_became_peered":"2020-10-09T17:14:33.164513+0000","last_unstale":"2020-10-09T17:14:33.167112+0000","last_undegraded":"2020-10-09T17:14:33.167112+0000","last_fullsized":"2020-10-09T17:14:33.167112+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.165530+0000","last_change":"2020-10-09T17:14:33.163879+0000","last_active":"2020-10-09T17:14:33.165530+0000","last_peered":"2020-10-09T17:14:33.165530+0000","last_clean":"2020-10-09T17:14:33.165530+0000","last_became_active":"2020-10-09T17:14:33.163371+0000","last_became_peered":"2020-10-09T17:14:33.163371+0000","last_unstale":"2020-10-09T17:14:33.165530+0000","last_undegraded":"2020-10-09T17:14:33.165530+0000","last_fullsized":"2020-10-09T17:14:33.165530+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167589+0000","last_change":"2020-10-09T17:14:33.165734+0000","last_active":"2020-10-09T17:14:33.167589+0000","last_peered":"2020-10-09T17:14:33.167589+0000","last_clean":"2020-10-09T17:14:33.167589+0000","last_became_active":"2020-10-09T17:14:33.164676+0000","last_became_peered":"2020-10-09T17:14:33.164676+0000","last_unstale":"2020-10-09T17:14:33.167589+0000","last_undegraded":"2020-10-09T17:14:33.167589+0000","last_fullsized":"2020-10-09T17:14:33.167589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.162459+0000","last_change":"2020-10-09T17:14:33.162459+0000","last_active":"2020-10-09T17:14:33.162459+0000","last_peered":"2020-10-09T17:14:33.162459+0000","last_clean":"2020-10-09T17:14:33.162459+0000","last_became_active":"2020-10-09T17:14:33.161998+0000","last_became_peered":"2020-10-09T17:14:33.161998+0000","last_unstale":"2020-10-09T17:14:33.162459+0000","last_undegraded":"2020-10-09T17:14:33.162459+0000","last_fullsized":"2020-10-09T17:14:33.162459+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.159728+0000","last_change":"2020-10-09T17:14:29.846060+0000","last_active":"2020-10-09T17:14:33.159728+0000","last_peered":"2020-10-09T17:14:33.159728+0000","last_clean":"2020-10-09T17:14:33.159728+0000","last_became_active":"2020-10-09T17:14:29.844447+0000","last_became_peered":"2020-10-09T17:14:29.844447+0000","last_unstale":"2020-10-09T17:14:33.159728+0000","last_undegraded":"2020-10-09T17:14:33.159728+0000","last_fullsized":"2020-10-09T17:14:33.159728+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:26.985932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:26.985932+0000","last_clean_scrub_stamp":"2020-10-09T17:14:26.985932+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.164769+0000","last_change":"2020-10-09T17:14:33.162406+0000","last_active":"2020-10-09T17:14:33.164769+0000","last_peered":"2020-10-09T17:14:33.164769+0000","last_clean":"2020-10-09T17:14:33.164769+0000","last_became_active":"2020-10-09T17:14:33.161733+0000","last_became_peered":"2020-10-09T17:14:33.161733+0000","last_unstale":"2020-10-09T17:14:33.164769+0000","last_undegraded":"2020-10-09T17:14:33.164769+0000","last_fullsized":"2020-10-09T17:14:33.164769+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.164263+0000","last_change":"2020-10-09T17:14:33.164263+0000","last_active":"2020-10-09T17:14:33.164263+0000","last_peered":"2020-10-09T17:14:33.164263+0000","last_clean":"2020-10-09T17:14:33.164263+0000","last_became_active":"2020-10-09T17:14:33.163679+0000","last_became_peered":"2020-10-09T17:14:33.163679+0000","last_unstale":"2020-10-09T17:14:33.164263+0000","last_undegraded":"2020-10-09T17:14:33.164263+0000","last_fullsized":"2020-10-09T17:14:33.164263+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167268+0000","last_change":"2020-10-09T17:14:33.167268+0000","last_active":"2020-10-09T17:14:33.167268+0000","last_peered":"2020-10-09T17:14:33.167268+0000","last_clean":"2020-10-09T17:14:33.167268+0000","last_became_active":"2020-10-09T17:14:33.166825+0000","last_became_peered":"2020-10-09T17:14:33.166825+0000","last_unstale":"2020-10-09T17:14:33.167268+0000","last_undegraded":"2020-10-09T17:14:33.167268+0000","last_fullsized":"2020-10-09T17:14:33.167268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.165744+0000","last_change":"2020-10-09T17:14:33.163939+0000","last_active":"2020-10-09T17:14:33.165744+0000","last_peered":"2020-10-09T17:14:33.165744+0000","last_clean":"2020-10-09T17:14:33.165744+0000","last_became_active":"2020-10-09T17:14:33.163628+0000","last_became_peered":"2020-10-09T17:14:33.163628+0000","last_unstale":"2020-10-09T17:14:33.165744+0000","last_undegraded":"2020-10-09T17:14:33.165744+0000","last_fullsized":"2020-10-09T17:14:33.165744+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]}]}]}} 2020-10-09T17:14:39.328 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-10-09T17:14:39.610 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:39.612 INFO:teuthology.orchestra.run.smithi184.stderr:dumped all 2020-10-09T17:14:39.623 INFO:teuthology.orchestra.run.smithi184.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-10-09T17:14:38.651773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":14680,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":14336,"kb_avail":314558120,"statfs":{"total":322122547200,"available":322107514880,"internally_reserved":0,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14680064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.494978"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167112+0000","last_change":"2020-10-09T17:14:33.165329+0000","last_active":"2020-10-09T17:14:33.167112+0000","last_peered":"2020-10-09T17:14:33.167112+0000","last_clean":"2020-10-09T17:14:33.167112+0000","last_became_active":"2020-10-09T17:14:33.164513+0000","last_became_peered":"2020-10-09T17:14:33.164513+0000","last_unstale":"2020-10-09T17:14:33.167112+0000","last_undegraded":"2020-10-09T17:14:33.167112+0000","last_fullsized":"2020-10-09T17:14:33.167112+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.165530+0000","last_change":"2020-10-09T17:14:33.163879+0000","last_active":"2020-10-09T17:14:33.165530+0000","last_peered":"2020-10-09T17:14:33.165530+0000","last_clean":"2020-10-09T17:14:33.165530+0000","last_became_active":"2020-10-09T17:14:33.163371+0000","last_became_peered":"2020-10-09T17:14:33.163371+0000","last_unstale":"2020-10-09T17:14:33.165530+0000","last_undegraded":"2020-10-09T17:14:33.165530+0000","last_fullsized":"2020-10-09T17:14:33.165530+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167589+0000","last_change":"2020-10-09T17:14:33.165734+0000","last_active":"2020-10-09T17:14:33.167589+0000","last_peered":"2020-10-09T17:14:33.167589+0000","last_clean":"2020-10-09T17:14:33.167589+0000","last_became_active":"2020-10-09T17:14:33.164676+0000","last_became_peered":"2020-10-09T17:14:33.164676+0000","last_unstale":"2020-10-09T17:14:33.167589+0000","last_undegraded":"2020-10-09T17:14:33.167589+0000","last_fullsized":"2020-10-09T17:14:33.167589+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.162459+0000","last_change":"2020-10-09T17:14:33.162459+0000","last_active":"2020-10-09T17:14:33.162459+0000","last_peered":"2020-10-09T17:14:33.162459+0000","last_clean":"2020-10-09T17:14:33.162459+0000","last_became_active":"2020-10-09T17:14:33.161998+0000","last_became_peered":"2020-10-09T17:14:33.161998+0000","last_unstale":"2020-10-09T17:14:33.162459+0000","last_undegraded":"2020-10-09T17:14:33.162459+0000","last_fullsized":"2020-10-09T17:14:33.162459+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.159728+0000","last_change":"2020-10-09T17:14:29.846060+0000","last_active":"2020-10-09T17:14:33.159728+0000","last_peered":"2020-10-09T17:14:33.159728+0000","last_clean":"2020-10-09T17:14:33.159728+0000","last_became_active":"2020-10-09T17:14:29.844447+0000","last_became_peered":"2020-10-09T17:14:29.844447+0000","last_unstale":"2020-10-09T17:14:33.159728+0000","last_undegraded":"2020-10-09T17:14:33.159728+0000","last_fullsized":"2020-10-09T17:14:33.159728+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:26.985932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:26.985932+0000","last_clean_scrub_stamp":"2020-10-09T17:14:26.985932+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.164769+0000","last_change":"2020-10-09T17:14:33.162406+0000","last_active":"2020-10-09T17:14:33.164769+0000","last_peered":"2020-10-09T17:14:33.164769+0000","last_clean":"2020-10-09T17:14:33.164769+0000","last_became_active":"2020-10-09T17:14:33.161733+0000","last_became_peered":"2020-10-09T17:14:33.161733+0000","last_unstale":"2020-10-09T17:14:33.164769+0000","last_undegraded":"2020-10-09T17:14:33.164769+0000","last_fullsized":"2020-10-09T17:14:33.164769+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.164263+0000","last_change":"2020-10-09T17:14:33.164263+0000","last_active":"2020-10-09T17:14:33.164263+0000","last_peered":"2020-10-09T17:14:33.164263+0000","last_clean":"2020-10-09T17:14:33.164263+0000","last_became_active":"2020-10-09T17:14:33.163679+0000","last_became_peered":"2020-10-09T17:14:33.163679+0000","last_unstale":"2020-10-09T17:14:33.164263+0000","last_undegraded":"2020-10-09T17:14:33.164263+0000","last_fullsized":"2020-10-09T17:14:33.164263+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.167268+0000","last_change":"2020-10-09T17:14:33.167268+0000","last_active":"2020-10-09T17:14:33.167268+0000","last_peered":"2020-10-09T17:14:33.167268+0000","last_clean":"2020-10-09T17:14:33.167268+0000","last_became_active":"2020-10-09T17:14:33.166825+0000","last_became_peered":"2020-10-09T17:14:33.166825+0000","last_unstale":"2020-10-09T17:14:33.167268+0000","last_undegraded":"2020-10-09T17:14:33.167268+0000","last_fullsized":"2020-10-09T17:14:33.167268+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-10-09T17:14:33.165744+0000","last_change":"2020-10-09T17:14:33.163939+0000","last_active":"2020-10-09T17:14:33.165744+0000","last_peered":"2020-10-09T17:14:33.165744+0000","last_clean":"2020-10-09T17:14:33.165744+0000","last_became_active":"2020-10-09T17:14:33.163628+0000","last_became_peered":"2020-10-09T17:14:33.163628+0000","last_unstale":"2020-10-09T17:14:33.165744+0000","last_undegraded":"2020-10-09T17:14:33.165744+0000","last_fullsized":"2020-10-09T17:14:33.165744+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-09T17:14:32.142630+0000","last_clean_scrub_stamp":"2020-10-09T17:14:32.142630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4848,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4736,"kb_avail":104852752,"statfs":{"total":107374182400,"available":107369218048,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4849664},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.304}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852688,"statfs":{"total":107374182400,"available":107369152512,"internally_reserved":0,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2669999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":104852680,"statfs":{"total":107374182400,"available":107369144320,"internally_reserved":0,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7909999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6259999999999999}]}]}]}} 2020-10-09T17:14:39.625 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-10-09T17:14:39.625 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-10-09T17:14:39.625 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-10-09T17:14:39.625 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-10-09T17:14:39.961 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:14:39.975 INFO:teuthology.orchestra.run.smithi184.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-10-09T17:14:39.976 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-10-09T17:14:39.976 INFO:teuthology.run_tasks:Running task rgw... 2020-10-09T17:14:39.989 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-10-09T17:14:39.990 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-10-09T17:14:39.990 INFO:tasks.rgw:Creating data pools 2020-10-09T17:14:39.991 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-10-09T17:14:39.991 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-10-09T17:14:40.654 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.data' created 2020-10-09T17:14:40.669 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-10-09T17:14:41.675 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-09T17:14:41.696 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-10-09T17:14:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.index' created 2020-10-09T17:14:42.690 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-10-09T17:14:43.708 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-09T17:14:43.732 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-10-09T17:14:43.732 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-10-09T17:14:44.011 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.data' already exists 2020-10-09T17:14:44.022 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-10-09T17:14:44.683 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-09T17:14:44.699 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-10-09T17:14:44.999 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.index' already exists 2020-10-09T17:14:45.010 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-10-09T17:14:45.687 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-09T17:14:45.701 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-10-09T17:14:45.702 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-10-09T17:14:46.994 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' created 2020-10-09T17:14:47.008 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-10-09T17:14:48.003 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-09T17:14:48.030 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-10-09T17:14:48.998 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' created 2020-10-09T17:14:49.022 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-10-09T17:14:50.030 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-09T17:14:50.053 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-10-09T17:14:50.054 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-10-09T17:14:50.332 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.data' already exists 2020-10-09T17:14:50.343 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-10-09T17:14:51.013 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-10-09T17:14:51.026 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-10-09T17:14:51.309 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'default.rgw.buckets.index' already exists 2020-10-09T17:14:51.320 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-10-09T17:14:52.016 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-10-09T17:14:52.028 DEBUG:tasks.rgw:Pools created 2020-10-09T17:14:52.028 INFO:tasks.rgw:Configuring compression type = random 2020-10-09T17:14:52.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-10-09T17:14:52.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-10-09T17:14:52.030 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-10-09T17:14:52.065 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:14:52.065 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:14:52.078 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:52.074+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.647 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.642+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.648 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.642+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.656 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.654+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.657 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.654+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.657 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.654+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.658 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.654+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.663 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.658+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.680 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.681 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.682 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 10 cannot find current period zonegroup using local zonegroup 2020-10-09T17:14:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.683 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 10 Creating default zonegroup 2020-10-09T17:14:53.684 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.678+0000 7f662e4f90c0 20 get_system_obj_state: rctx=0x7ffceca33130 obj=:.pools.avail state=0x55bcc6cf52e0 s->prefetch_data=0 2020-10-09T17:14:53.685 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.682+0000 7f662e4f90c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-10-09T17:14:53.704 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.702+0000 7f662e4f90c0 10 WARNING: store->list_zones() returned r=-2 2020-10-09T17:14:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.702+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.702+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.710 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.706+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.706+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.706+0000 7f662e4f90c0 10 could not read realm id: (2) No such file or directory 2020-10-09T17:14:53.711 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.706+0000 7f662e4f90c0 10 WARNING: failed to set zone as default, r=-22 2020-10-09T17:14:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.706+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.712 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.710+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.717 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.718 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:14:53.718 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.719 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:14:53.719 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 zonegroup default 2020-10-09T17:14:53.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.714+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:53.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 10 Cannot find current period zone using local zone 2020-10-09T17:14:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 10 Using default name default 2020-10-09T17:14:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:14:53.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:53.722 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:14:53.722 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 zone default found 2020-10-09T17:14:53.722 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:53.718+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:55.673 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:55.670+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:55.673 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:55.670+0000 7f662e4f90c0 20 rados->read ofs=0 len=0 2020-10-09T17:14:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:55.670+0000 7f662e4f90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:14:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:55.670+0000 7f662e4f90c0 20 started sync module instance, tier type = 2020-10-09T17:14:55.674 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:55.670+0000 7f662e4f90c0 20 started zone id=dbcd1769-f23f-49d0-86ac-d7356ddc50c8 (name=default) with tier type = 2020-10-09T17:14:57.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=0 2020-10-09T17:14:57.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=1 2020-10-09T17:14:57.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=2 2020-10-09T17:14:57.720 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=3 2020-10-09T17:14:57.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=4 2020-10-09T17:14:57.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=5 2020-10-09T17:14:57.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.714+0000 7f662e4f90c0 20 add_watcher() i=6 2020-10-09T17:14:57.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.718+0000 7f662e4f90c0 20 add_watcher() i=7 2020-10-09T17:14:57.721 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.718+0000 7f662e4f90c0 2 all 8 watchers are set, enabling cache 2020-10-09T17:14:57.734 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.730+0000 7f65ebfdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:14:57.734 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.730+0000 7f662e4f90c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:14:57.735 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:14:57.730+0000 7f662e4f90c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:01.679 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.674+0000 7f662e4f90c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-10-09T17:15:01.791 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f655affd700 20 reqs_thread_entry: start 2020-10-09T17:15:01.791 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f6558ff9700 20 reqs_thread_entry: start 2020-10-09T17:15:01.791 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f662e4f90c0 20 init_complete bucket index max shards: 11 2020-10-09T17:15:01.792 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f662e4f90c0 10 Started notification manager with: 1 workers 2020-10-09T17:15:01.792 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f653effd700 20 INFO: next queues processing will happen at: Fri Oct 9 17:15:32 2020 2020-10-09T17:15:01.792 INFO:teuthology.orchestra.run.smithi060.stderr: 2020-10-09T17:15:01.792 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.786+0000 7f653e7fc700 20 reqs_thread_entry: start 2020-10-09T17:15:01.822 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:15:01.823 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.818+0000 7f662e4f90c0 20 remove_watcher() i=0 2020-10-09T17:15:01.823 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.818+0000 7f662e4f90c0 2 removed watcher, disabling cache 2020-10-09T17:15:01.825 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.822+0000 7f662e4f90c0 20 remove_watcher() i=1 2020-10-09T17:15:01.828 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.826+0000 7f662e4f90c0 20 remove_watcher() i=2 2020-10-09T17:15:01.831 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.826+0000 7f662e4f90c0 20 remove_watcher() i=3 2020-10-09T17:15:01.834 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.830+0000 7f662e4f90c0 20 remove_watcher() i=4 2020-10-09T17:15:01.837 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.834+0000 7f662e4f90c0 20 remove_watcher() i=5 2020-10-09T17:15:01.840 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.838+0000 7f662e4f90c0 20 remove_watcher() i=6 2020-10-09T17:15:01.843 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.838+0000 7f662e4f90c0 20 remove_watcher() i=7 2020-10-09T17:15:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.842+0000 7f65ebfdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:01.859 DEBUG:tasks.util.rgw: json result: [] 2020-10-09T17:15:01.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:01.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:01.861 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-09T17:15:01.942 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:01.943 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.950+0000 7faf57ada0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:01.955 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.950+0000 7faf57ada0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faec57fa700 20 reqs_thread_entry: start 2020-10-09T17:15:01.964 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:01.965 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:01.965 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:15:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:01.967 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.962+0000 7faf57ada0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.966+0000 7faf57ada0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.966+0000 7faf57ada0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:01.969 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:01.966+0000 7faf57ada0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:01.976 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:01.976 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "dbcd1769-f23f-49d0-86ac-d7356ddc50c8", 2020-10-09T17:15:01.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "default.rgw.meta:root", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "default.rgw.control", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "default.rgw.log", 2020-10-09T17:15:01.977 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-09T17:15:01.978 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "default.rgw.otp", 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "", 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "" 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:01.979 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:01.980 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: "compression_type": "random" 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:01.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:01.982 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:01.982 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "", 2020-10-09T17:15:01.982 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-09T17:15:01.982 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:01.983 DEBUG:tasks.util.rgw: json result: {'id': 'dbcd1769-f23f-49d0-86ac-d7356ddc50c8', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-09T17:15:01.983 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-10-09T17:15:01.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-10-09T17:15:01.984 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-10-09T17:15:02.062 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:02.062 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:02.071 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.066+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.075 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.070+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.075 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.070+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.075 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.070+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.076 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.070+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.076 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.074+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.076 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.074+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.077 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.074+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.104 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 RGWRados::pool_iterate: got zone_info.dbcd1769-f23f-49d0-86ac-d7356ddc50c8 2020-10-09T17:15:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 RGWRados::pool_iterate: got zonegroup_info.4f71624a-46a1-45f4-a14b-41f6ee4e8781 2020-10-09T17:15:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 RGWRados::pool_iterate: got zone_names.default 2020-10-09T17:15:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-10-09T17:15:02.105 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:02.106 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.107 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.107 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.102+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.107 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.108 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 10 cannot find current period zonegroup using local zonegroup 2020-10-09T17:15:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.109 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.110 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.110 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.110 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:02.110 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 zonegroup default 2020-10-09T17:15:02.111 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.111 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.111 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 10 Cannot find current period zone using local zone 2020-10-09T17:15:02.111 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.106+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.113 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:02.113 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 zone default found 2020-10-09T17:15:02.113 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 started sync module instance, tier type = 2020-10-09T17:15:02.115 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.110+0000 7f92dbad40c0 20 started zone id=dbcd1769-f23f-49d0-86ac-d7356ddc50c8 (name=default) with tier type = 2020-10-09T17:15:02.136 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=0 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=1 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=2 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=3 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=4 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=5 2020-10-09T17:15:02.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=6 2020-10-09T17:15:02.138 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 20 add_watcher() i=7 2020-10-09T17:15:02.138 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.134+0000 7f92dbad40c0 2 all 8 watchers are set, enabling cache 2020-10-09T17:15:02.157 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.154+0000 7f92997e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:02.157 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.154+0000 7f92dbad40c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:02.157 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.154+0000 7f92dbad40c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.246+0000 7f9207fff700 20 reqs_thread_entry: start 2020-10-09T17:15:02.252 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.246+0000 7f9205ffb700 20 reqs_thread_entry: start 2020-10-09T17:15:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.246+0000 7f92dbad40c0 20 init_complete bucket index max shards: 11 2020-10-09T17:15:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.246+0000 7f92dbad40c0 10 Started notification manager with: 1 workers 2020-10-09T17:15:02.253 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.250+0000 7f91f3fff700 20 INFO: next queues processing will happen at: Fri Oct 9 17:15:32 2020 2020-10-09T17:15:02.253 INFO:teuthology.orchestra.run.smithi060.stderr: 2020-10-09T17:15:02.254 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.250+0000 7f91f37fe700 20 reqs_thread_entry: start 2020-10-09T17:15:02.279 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:15:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.274+0000 7f92dbad40c0 20 remove_watcher() i=0 2020-10-09T17:15:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.274+0000 7f92dbad40c0 2 removed watcher, disabling cache 2020-10-09T17:15:02.282 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.278+0000 7f92dbad40c0 20 remove_watcher() i=1 2020-10-09T17:15:02.285 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.282+0000 7f92dbad40c0 20 remove_watcher() i=2 2020-10-09T17:15:02.288 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.286+0000 7f92dbad40c0 20 remove_watcher() i=3 2020-10-09T17:15:02.291 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.286+0000 7f92dbad40c0 20 remove_watcher() i=4 2020-10-09T17:15:02.294 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.290+0000 7f92dbad40c0 20 remove_watcher() i=5 2020-10-09T17:15:02.296 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.294+0000 7f92dbad40c0 20 remove_watcher() i=6 2020-10-09T17:15:02.299 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.294+0000 7f92dbad40c0 20 remove_watcher() i=7 2020-10-09T17:15:02.301 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.298+0000 7f92997e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:02.307 DEBUG:tasks.util.rgw: json result: [] 2020-10-09T17:15:02.308 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:02.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:02.308 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-09T17:15:02.391 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:02.392 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:02.405 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.402+0000 7fe24250e0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:02.406 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.402+0000 7fe24250e0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:02.412 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe177fff700 20 reqs_thread_entry: start 2020-10-09T17:15:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe24250e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.413 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe24250e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.414 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe24250e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.415 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe24250e0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.410+0000 7fe24250e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.414+0000 7fe24250e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.414+0000 7fe24250e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.416 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.414+0000 7fe24250e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:02.417 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.414+0000 7fe24250e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:02.417 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:02.414+0000 7fe24250e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:02.425 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "dbcd1769-f23f-49d0-86ac-d7356ddc50c8", 2020-10-09T17:15:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default", 2020-10-09T17:15:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "default.rgw.meta:root", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "default.rgw.control", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "default.rgw.log", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-09T17:15:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-09T17:15:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "default.rgw.otp", 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "", 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "" 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "compression_type": "random" 2020-10-09T17:15:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "", 2020-10-09T17:15:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-09T17:15:02.431 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:02.432 DEBUG:tasks.util.rgw: json result: {'id': 'dbcd1769-f23f-49d0-86ac-d7356ddc50c8', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-09T17:15:02.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-10-09T17:15:02.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-10-09T17:15:02.433 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-10-09T17:15:02.467 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:02.468 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:02.480 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:02.482+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.754 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.754+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.754 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.754+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.754 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.754+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.755 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.754+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.758 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.758+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.758 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.758+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.759 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.758+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.775 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.774+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.776 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.776 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.776 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.776 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.777 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.777 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 10 cannot find current period zonegroup using local zonegroup 2020-10-09T17:15:03.777 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.777 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 10 Creating default zonegroup 2020-10-09T17:15:03.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 20 get_system_obj_state: rctx=0x7fffdc64e9f0 obj=:.pools.avail state=0x558c7a6bb2e0 s->prefetch_data=0 2020-10-09T17:15:03.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.778+0000 7f0f910ce0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-10-09T17:15:03.793 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.794+0000 7f0f910ce0c0 10 WARNING: store->list_zones() returned r=-2 2020-10-09T17:15:03.793 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.794+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.794 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.794+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.799 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.798+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.799 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.802+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.799 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.802+0000 7f0f910ce0c0 10 could not read realm id: (2) No such file or directory 2020-10-09T17:15:03.800 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.802+0000 7f0f910ce0c0 10 WARNING: failed to set zone as default, r=-22 2020-10-09T17:15:03.800 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.802+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.801 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.802+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.805 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.806+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.806 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.806+0000 7f0f910ce0c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:15:03.806 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.806+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.807 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:03.808 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 zonegroup default 2020-10-09T17:15:03.808 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.809 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:03.809 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 10 Cannot find current period zone using local zone 2020-10-09T17:15:03.809 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 10 Using default name default 2020-10-09T17:15:03.809 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.810 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:03.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:03.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:15:03.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 zone default found 2020-10-09T17:15:03.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:03.810+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:05.777 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:05.778+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:05.778 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:05.778+0000 7f0f910ce0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:05.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:05.782+0000 7f0f910ce0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:05.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:05.782+0000 7f0f910ce0c0 20 started sync module instance, tier type = 2020-10-09T17:15:05.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:05.782+0000 7f0f910ce0c0 20 started zone id=577eb5f0-bfb4-4482-a09f-af137aa06711 (name=default) with tier type = 2020-10-09T17:15:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=0 2020-10-09T17:15:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=1 2020-10-09T17:15:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=2 2020-10-09T17:15:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=3 2020-10-09T17:15:07.817 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=4 2020-10-09T17:15:07.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=5 2020-10-09T17:15:07.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=6 2020-10-09T17:15:07.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 20 add_watcher() i=7 2020-10-09T17:15:07.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.818+0000 7f0f910ce0c0 2 all 8 watchers are set, enabling cache 2020-10-09T17:15:07.835 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.838+0000 7f0f4e7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:07.835 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.838+0000 7f0f910ce0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:07.836 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:07.838+0000 7f0f910ce0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:11.804 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.805+0000 7f0f910ce0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-10-09T17:15:11.910 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0f257fa700 20 reqs_thread_entry: start 2020-10-09T17:15:11.911 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0f910ce0c0 20 init_complete bucket index max shards: 11 2020-10-09T17:15:11.911 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0eaaffd700 20 reqs_thread_entry: start 2020-10-09T17:15:11.912 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0f910ce0c0 10 Started notification manager with: 1 workers 2020-10-09T17:15:11.912 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0ea97fa700 20 INFO: next queues processing will happen at: Fri Oct 9 17:15:42 2020 2020-10-09T17:15:11.912 INFO:teuthology.orchestra.run.smithi184.stderr: 2020-10-09T17:15:11.913 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.909+0000 7f0ea8ff9700 20 reqs_thread_entry: start 2020-10-09T17:15:11.941 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:15:11.941 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.941+0000 7f0f910ce0c0 20 remove_watcher() i=0 2020-10-09T17:15:11.941 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.941+0000 7f0f910ce0c0 2 removed watcher, disabling cache 2020-10-09T17:15:11.943 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.941+0000 7f0f910ce0c0 20 remove_watcher() i=1 2020-10-09T17:15:11.946 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.945+0000 7f0f910ce0c0 20 remove_watcher() i=2 2020-10-09T17:15:11.948 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.949+0000 7f0f910ce0c0 20 remove_watcher() i=3 2020-10-09T17:15:11.951 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.949+0000 7f0f910ce0c0 20 remove_watcher() i=4 2020-10-09T17:15:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.953+0000 7f0f910ce0c0 20 remove_watcher() i=5 2020-10-09T17:15:11.956 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.957+0000 7f0f910ce0c0 20 remove_watcher() i=6 2020-10-09T17:15:11.959 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.957+0000 7f0f910ce0c0 20 remove_watcher() i=7 2020-10-09T17:15:11.960 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:11.961+0000 7f0f4e7e4700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:11.975 DEBUG:tasks.util.rgw: json result: [] 2020-10-09T17:15:11.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:11.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:11.976 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-09T17:15:12.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:12.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:12.065 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.065+0000 7f5d7745a0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:12.066 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.065+0000 7f5d7745a0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:12.073 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5cabfff700 20 reqs_thread_entry: start 2020-10-09T17:15:12.073 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.074 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.074 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.075 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados_obj.operate() r=0 bl.length=873 2020-10-09T17:15:12.075 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.075 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.076 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.076 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.076 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.073+0000 7f5d7745a0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.077 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.077+0000 7f5d7745a0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:12.084 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:12.084 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "577eb5f0-bfb4-4482-a09f-af137aa06711", 2020-10-09T17:15:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2020-10-09T17:15:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2020-10-09T17:15:12.085 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2020-10-09T17:15:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-09T17:15:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-09T17:15:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2020-10-09T17:15:12.086 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-09T17:15:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-09T17:15:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-09T17:15:12.087 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-09T17:15:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-09T17:15:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-09T17:15:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-09T17:15:12.088 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-09T17:15:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2020-10-09T17:15:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:15:12.089 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2020-10-09T17:15:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2020-10-09T17:15:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:15:12.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:15:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:15:12.091 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-09T17:15:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:15:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:15:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-09T17:15:12.092 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:15:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:12.093 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-09T17:15:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:15:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.094 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-10-09T17:15:12.095 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-09T17:15:12.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:12.097 DEBUG:tasks.util.rgw: json result: {'id': '577eb5f0-bfb4-4482-a09f-af137aa06711', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-09T17:15:12.097 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-10-09T17:15:12.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-10-09T17:15:12.098 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-10-09T17:15:12.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:12.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:12.189 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.189+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.191 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.189+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.192 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.189+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.192 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.193+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.193 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.193+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.193 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.193+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.194 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.193+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.194 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.193+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.217 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 RGWRados::pool_iterate: got zonegroup_info.bd8620ec-0163-46d8-bd93-36e516348453 2020-10-09T17:15:12.217 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 RGWRados::pool_iterate: got zone_info.577eb5f0-bfb4-4482-a09f-af137aa06711 2020-10-09T17:15:12.217 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 RGWRados::pool_iterate: got zone_names.default 2020-10-09T17:15:12.218 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-10-09T17:15:12.218 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.218 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.218 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.219 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:12.219 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.219 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.219 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.220 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:12.220 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.217+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.221 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.221 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 10 cannot find current period zonegroup using local zonegroup 2020-10-09T17:15:12.221 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.221 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.222 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.222 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.222 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.223 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:12.224 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 zonegroup default 2020-10-09T17:15:12.224 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.224 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 10 Cannot find current period zone using local zone 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.221+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.225 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.226 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:12.226 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 zone default found 2020-10-09T17:15:12.226 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.226 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.227 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.227 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.227 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 started sync module instance, tier type = 2020-10-09T17:15:12.227 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.225+0000 7fb852be90c0 20 started zone id=577eb5f0-bfb4-4482-a09f-af137aa06711 (name=default) with tier type = 2020-10-09T17:15:12.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=0 2020-10-09T17:15:12.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=1 2020-10-09T17:15:12.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=2 2020-10-09T17:15:12.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=3 2020-10-09T17:15:12.261 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=4 2020-10-09T17:15:12.261 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=5 2020-10-09T17:15:12.261 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.257+0000 7fb852be90c0 20 add_watcher() i=6 2020-10-09T17:15:12.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.261+0000 7fb852be90c0 20 add_watcher() i=7 2020-10-09T17:15:12.262 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.261+0000 7fb852be90c0 2 all 8 watchers are set, enabling cache 2020-10-09T17:15:12.281 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.281+0000 7fb80f7ee700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:12.282 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.281+0000 7fb852be90c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:12.282 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.281+0000 7fb852be90c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:12.373 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb77e7fc700 20 reqs_thread_entry: start 2020-10-09T17:15:12.374 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb76ffff700 20 reqs_thread_entry: start 2020-10-09T17:15:12.374 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb852be90c0 20 init_complete bucket index max shards: 11 2020-10-09T17:15:12.375 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb852be90c0 10 Started notification manager with: 1 workers 2020-10-09T17:15:12.375 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb76e7fc700 20 INFO: next queues processing will happen at: Fri Oct 9 17:15:42 2020 2020-10-09T17:15:12.375 INFO:teuthology.orchestra.run.smithi184.stderr: 2020-10-09T17:15:12.375 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.373+0000 7fb76dffb700 20 reqs_thread_entry: start 2020-10-09T17:15:12.401 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:15:12.401 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.401+0000 7fb852be90c0 20 remove_watcher() i=0 2020-10-09T17:15:12.402 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.401+0000 7fb852be90c0 2 removed watcher, disabling cache 2020-10-09T17:15:12.404 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.401+0000 7fb852be90c0 20 remove_watcher() i=1 2020-10-09T17:15:12.406 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.405+0000 7fb852be90c0 20 remove_watcher() i=2 2020-10-09T17:15:12.409 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.409+0000 7fb852be90c0 20 remove_watcher() i=3 2020-10-09T17:15:12.412 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.413+0000 7fb852be90c0 20 remove_watcher() i=4 2020-10-09T17:15:12.415 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.413+0000 7fb852be90c0 20 remove_watcher() i=5 2020-10-09T17:15:12.418 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.417+0000 7fb852be90c0 20 remove_watcher() i=6 2020-10-09T17:15:12.420 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.421+0000 7fb852be90c0 20 remove_watcher() i=7 2020-10-09T17:15:12.421 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.421+0000 7fb80f7ee700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-10-09T17:15:12.427 DEBUG:tasks.util.rgw: json result: [] 2020-10-09T17:15:12.428 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:12.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-10-09T17:15:12.428 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-10-09T17:15:12.509 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:12.509 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:12.519 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.517+0000 7fbad007e0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-10-09T17:15:12.519 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.517+0000 7fbad007e0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-10-09T17:15:12.523 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.521+0000 7fba157fa700 20 reqs_thread_entry: start 2020-10-09T17:15:12.524 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.521+0000 7fbad007e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.526 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.526 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.527 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados_obj.operate() r=0 bl.length=889 2020-10-09T17:15:12.527 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.528 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-10-09T17:15:12.528 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.525+0000 7fbad007e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.529 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.529+0000 7fbad007e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-10-09T17:15:12.529 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.529+0000 7fbad007e0c0 20 rados->read ofs=0 len=0 2020-10-09T17:15:12.530 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:12.529+0000 7fbad007e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-10-09T17:15:12.536 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:12.536 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "577eb5f0-bfb4-4482-a09f-af137aa06711", 2020-10-09T17:15:12.536 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default", 2020-10-09T17:15:12.537 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "default.rgw.meta:root", 2020-10-09T17:15:12.537 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "default.rgw.control", 2020-10-09T17:15:12.537 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "default.rgw.log:gc", 2020-10-09T17:15:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "default.rgw.log:lc", 2020-10-09T17:15:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "default.rgw.log", 2020-10-09T17:15:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "default.rgw.log:intent", 2020-10-09T17:15:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "default.rgw.log:usage", 2020-10-09T17:15:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "default.rgw.meta:roles", 2020-10-09T17:15:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "default.rgw.log:reshard", 2020-10-09T17:15:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2020-10-09T17:15:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "default.rgw.meta:users.email", 2020-10-09T17:15:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2020-10-09T17:15:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2020-10-09T17:15:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "default.rgw.otp", 2020-10-09T17:15:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:15:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "", 2020-10-09T17:15:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "" 2020-10-09T17:15:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:15:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:15:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:15:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "default.rgw.buckets.index", 2020-10-09T17:15:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:15:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:15:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "default.rgw.buckets.data", 2020-10-09T17:15:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:15:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2020-10-09T17:15:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:15:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:12.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-10-09T17:15:12.547 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "default.rgw.log:notif" 2020-10-09T17:15:12.547 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:12.548 DEBUG:tasks.util.rgw: json result: {'id': '577eb5f0-bfb4-4482-a09f-af137aa06711', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-10-09T17:15:12.549 INFO:tasks.rgw:Starting rgw... 2020-10-09T17:15:12.550 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-10-09T17:15:12.550 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-09T17:15:12.551 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-10-09T17:15:12.551 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-10-09T17:15:12.554 INFO:tasks.rgw.c1.client.0:Started 2020-10-09T17:15:12.555 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-10-09T17:15:12.555 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-09T17:15:12.556 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-09T17:15:12.556 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-09T17:15:12.559 INFO:tasks.rgw.c1.client.1:Started 2020-10-09T17:15:12.560 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-10-09T17:15:12.560 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-09T17:15:12.561 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-10-09T17:15:12.561 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-10-09T17:15:12.594 INFO:tasks.rgw.c2.client.0:Started 2020-10-09T17:15:12.595 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-10-09T17:15:12.595 INFO:tasks.rgw:Using beast as radosgw frontend 2020-10-09T17:15:12.596 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-10-09T17:15:12.596 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-10-09T17:15:12.599 INFO:tasks.rgw.c2.client.1:Started 2020-10-09T17:15:12.599 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi060.front.sepia.ceph.com:8000/ 2020-10-09T17:15:12.599 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000/ 2020-10-09T17:15:12.622 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:12.622 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:12.624 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8000: Connection refused 2020-10-09T17:15:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:15:13.626 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000/ 2020-10-09T17:15:13.643 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:13.644 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:13.648 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-09T17:15:13.649 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-10-09T17:15:13.649 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi060.front.sepia.ceph.com:8001/ 2020-10-09T17:15:13.650 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001/ 2020-10-09T17:15:13.704 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:13.704 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:13.708 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-10-09T17:15:13.709 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-10-09T17:15:13.709 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8000/ 2020-10-09T17:15:13.709 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000/ 2020-10-09T17:15:13.802 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:13.803 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:13.807 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 53500 2020-10-09T17:15:13.807 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-10-09T17:15:13.808 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi184.front.sepia.ceph.com:8001/ 2020-10-09T17:15:13.808 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001/ 2020-10-09T17:15:13.816 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:13.817 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:13.821 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2020-10-09T17:15:13.822 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-10-09T17:15:13.822 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-10-09T17:15:14.274 INFO:tasks.rgw_multisite:creating realm.. 2020-10-09T17:15:14.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.276 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:14.311 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:14.312 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:14.363 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-realm", 2020-10-09T17:15:14.364 INFO:teuthology.orchestra.run.smithi060.stdout: "current_period": "b12b9dd8-e452-43dd-8255-07a03ef04ce8", 2020-10-09T17:15:14.365 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 1 2020-10-09T17:15:14.365 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:14.366 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-10-09T17:15:14.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.367 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:14.446 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:14.465 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:14.464+0000 7f64ec7700c0 -1 old format 2020-10-09T17:15:14.480 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:14.480 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:15:14.480 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:14.481 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:15:14.482 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:15:14.483 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "", 2020-10-09T17:15:14.483 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [], 2020-10-09T17:15:14.483 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [], 2020-10-09T17:15:14.483 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "", 2020-10-09T17:15:14.484 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:14.484 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:15:14.484 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:15:14.484 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:14.484 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:14.485 INFO:tasks.rgw_multisite:creating master zone.. 2020-10-09T17:15:14.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:14.486 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --endpoints http://smithi060.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:14.574 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:14.575 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:14.645 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:14.646 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:14.646 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:15:14.646 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-09T17:15:14.646 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-09T17:15:14.647 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-09T17:15:14.648 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-09T17:15:14.648 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-09T17:15:14.648 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-09T17:15:14.648 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-09T17:15:14.648 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:14.649 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:14.650 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:14.651 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:14.652 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:14.652 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:14.652 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:14.652 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-09T17:15:14.652 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:14.653 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-10-09T17:15:15.663 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-10-09T17:15:15.677 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-10-09T17:15:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-10-09T17:15:16.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:16.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:16.690 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:16.736 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:16.760 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:16.760 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:16.761 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:15:16.761 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-09T17:15:16.761 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-09T17:15:16.761 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-09T17:15:16.762 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-09T17:15:16.762 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-09T17:15:16.762 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-09T17:15:16.762 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-09T17:15:16.762 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-09T17:15:16.763 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-09T17:15:16.763 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-09T17:15:16.763 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-09T17:15:16.763 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-09T17:15:16.763 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-09T17:15:16.764 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-09T17:15:16.764 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:16.764 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:16.764 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:16.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:16.765 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:16.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:16.765 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-10-09T17:15:16.766 INFO:teuthology.orchestra.run.smithi060.stdout: "compression_type": "random" 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:16.767 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "", 2020-10-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-09T17:15:16.768 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:16.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:16.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:16.770 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:16.852 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:16.852 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:24.900 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:24.901 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:15:24.901 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 1, 2020-10-09T17:15:24.902 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "b12b9dd8-e452-43dd-8255-07a03ef04ce8", 2020-10-09T17:15:24.902 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [], 2020-10-09T17:15:24.902 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:15:24.903 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:15:24.903 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:15:24.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:24.904 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:15:24.904 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:15:24.904 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:15:24.904 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:15:24.905 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:15:24.905 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:15:24.905 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:24.906 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:15:24.906 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:15:24.906 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:24.907 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:15:24.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:24.908 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:24.908 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:15:24.908 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:15:24.908 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "false", 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:15:24.909 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:15:24.910 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:15:24.910 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:15:24.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.911 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:24.911 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:15:24.911 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:24.912 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:15:24.912 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:15:24.912 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:15:24.912 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:15:24.913 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:15:24.913 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.913 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:24.914 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:15:24.914 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:24.914 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:15:24.915 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:15:24.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.915 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:24.916 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:15:24.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:24.916 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:24.917 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:15:24.917 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.917 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:15:24.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:24.918 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:15:24.918 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:24.918 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:15:24.919 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:15:24.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:24.920 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:15:24.920 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:15:24.920 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:15:24.920 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:15:24.921 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:15:24.921 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:15:24.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:24.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:24.922 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:24.922 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:15:24.923 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:15:24.923 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:24.925 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-10-09T17:15:24.925 INFO:teuthology.orchestra.run:waiting for 300 2020-10-09T17:15:24.966 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:15:24.964+0000 7f72a2b80700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 16704) UID: 0 2020-10-09T17:15:24.967 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:15:24.964+0000 7f72b9ae7100 -1 shutting down 2020-10-09T17:15:29.265 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-10-09T17:15:31.027 INFO:tasks.rgw.c1.client.0:Stopped 2020-10-09T17:15:31.028 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-10-09T17:15:31.028 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-10-09T17:15:31.031 INFO:tasks.rgw.c1.client.0:Started 2020-10-09T17:15:31.031 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000 2020-10-09T17:15:31.050 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:31.051 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:31.054 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8000: Connection refused 2020-10-09T17:15:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:15:32.056 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000 2020-10-09T17:15:32.074 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:32.074 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:32.079 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-09T17:15:32.079 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-10-09T17:15:32.080 INFO:tasks.rgw_multisite:creating admin user.. 2020-10-09T17:15:32.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.081 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:32.169 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:32.380 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:32.380 INFO:teuthology.orchestra.run.smithi060.stdout: "user_id": "realm-admin", 2020-10-09T17:15:32.380 INFO:teuthology.orchestra.run.smithi060.stdout: "display_name": "Realm Admin", 2020-10-09T17:15:32.380 INFO:teuthology.orchestra.run.smithi060.stdout: "email": "", 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "suspended": 0, 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "max_buckets": 1000, 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "subusers": [], 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "keys": [ 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "user": "realm-admin", 2020-10-09T17:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: "swift_keys": [], 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": [], 2020-10-09T17:15:32.382 INFO:teuthology.orchestra.run.smithi060.stdout: "op_mask": "read, write, delete", 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "system": "true", 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "", 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "default_storage_class": "", 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_tags": [], 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:15:32.383 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:15:32.384 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "temp_url_keys": [], 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "rgw", 2020-10-09T17:15:32.385 INFO:teuthology.orchestra.run.smithi060.stdout: "mfa_ids": [] 2020-10-09T17:15:32.386 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:32.386 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:15:32.410 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-10-09T17:15:32.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.411 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi060.front.sepia.ceph.com:8000 --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:32.456 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:32.456 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:32.578 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-realm", 2020-10-09T17:15:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:15:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2 2020-10-09T17:15:32.580 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:32.581 INFO:tasks.rgw_multisite:creating zone.. 2020-10-09T17:15:32.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:32.582 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --endpoints http://smithi184.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:32.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:32.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:32.730 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:32.726+0000 7f66851b40c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:15:32.731 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:15:32.726+0000 7f66851b40c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:15:32.742 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:15:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:15:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:15:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:15:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:15:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:15:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:15:32.747 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:32.749 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-10-09T17:15:34.033 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-10-09T17:15:34.047 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-10-09T17:15:34.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-10-09T17:15:35.060 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-10-09T17:15:35.085 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:35.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:35.085 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:35.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:35.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:35.147 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:35.147 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:15:35.147 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:15:35.148 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:15:35.149 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:35.150 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:15:35.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-10-09T17:15:35.152 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:15:35.153 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:35.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:35.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:35.154 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:35.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:35.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:43.248 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:15:43.346 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:15:43.346 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:15:43.346 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2, 2020-10-09T17:15:43.346 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "b12b9dd8-e452-43dd-8255-07a03ef04ce8", 2020-10-09T17:15:43.346 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:15:43.347 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.348 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:15:43.349 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:15:43.350 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:43.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:15:43.352 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:15:43.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:15:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:15:43.354 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.355 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.356 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:15:43.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:15:43.358 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:15:43.359 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:15:43.360 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:15:43.361 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:15:43.363 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-10-09T17:15:43.363 INFO:teuthology.orchestra.run:waiting for 300 2020-10-09T17:15:43.407 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:15:43.406+0000 7f678d520700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 16305) UID: 0 2020-10-09T17:15:43.407 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:15:43.406+0000 7f67a4487100 -1 shutting down 2020-10-09T17:15:46.329 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-10-09T17:15:49.465 INFO:tasks.rgw.c2.client.0:Stopped 2020-10-09T17:15:49.466 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-10-09T17:15:49.466 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-10-09T17:15:49.469 INFO:tasks.rgw.c2.client.0:Started 2020-10-09T17:15:49.469 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2020-10-09T17:15:49.485 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:49.486 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:49.489 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8000: Connection refused 2020-10-09T17:15:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:15:50.491 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8000 2020-10-09T17:15:50.508 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:15:50.508 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:15:50.513 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-09T17:15:50.514 INFO:teuthology.orchestra.run.smithi184.stdout:anonymous 2020-10-09T17:15:50.514 INFO:tasks.rgw_multisite:creating zone.. 2020-10-09T17:15:50.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:50.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:50.515 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --endpoints http://smithi060.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:50.628 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:50.624+0000 7f4eccdce0c0 0 failed reading obj info from .rgw.root:zone_info.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84: (2) No such file or directory 2020-10-09T17:15:50.628 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:15:50.624+0000 7f4eccdce0c0 0 WARNING: could not read zone params for zone id=bf2453fb-36ea-45e0-8abf-a48b7c4b1b84 name=test-zone2 2020-10-09T17:15:50.636 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:50.636 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:15:50.636 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:15:50.636 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "test-zone3.rgw.control", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "test-zone3.rgw.log", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-10-09T17:15:50.637 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-10-09T17:15:50.638 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:50.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2020-10-09T17:15:50.640 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:50.641 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:15:50.642 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-10-09T17:15:50.642 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:50.643 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-10-09T17:15:51.746 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-10-09T17:15:51.769 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-10-09T17:15:52.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-10-09T17:15:52.762 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-10-09T17:15:52.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:52.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:52.782 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:52.831 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:52.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:15:52.855 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:15:52.856 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:15:52.856 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:15:52.856 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "test-zone3.rgw.meta:root", 2020-10-09T17:15:52.856 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "test-zone3.rgw.control", 2020-10-09T17:15:52.856 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "test-zone3.rgw.log", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2020-10-09T17:15:52.857 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "test-zone3.rgw.otp", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:15:52.858 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2020-10-09T17:15:52.859 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: "compression_type": "random" 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2020-10-09T17:15:52.860 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "", 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2020-10-09T17:15:52.861 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:15:52.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:52.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:15:52.863 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:15:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:15:52.901 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:00.947 INFO:teuthology.orchestra.run.smithi060.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:16:01.009 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:16:01.009 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:01.009 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 3, 2020-10-09T17:16:01.009 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "b12b9dd8-e452-43dd-8255-07a03ef04ce8", 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [], 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:16:01.010 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:16:01.011 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:16:01.012 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:16:01.013 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:16:01.014 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:16:01.015 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:16:01.016 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:16:01.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.018 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.019 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:01.020 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 530799950 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:16:01.021 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 2021681268 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.022 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:16:01.023 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:16:01.024 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:16:01.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:16:01.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:16:01.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:16:01.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:01.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:01.037 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:01.037 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:16:01.037 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:16:01.037 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:16:01.038 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-10-09T17:16:01.039 INFO:teuthology.orchestra.run:waiting for 300 2020-10-09T17:16:01.085 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:16:01.084+0000 7f1342ffd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 16708) UID: 0 2020-10-09T17:16:01.086 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:16:01.084+0000 7f135e834100 -1 shutting down 2020-10-09T17:16:03.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-10-09T17:16:07.140 INFO:tasks.rgw.c1.client.1:Stopped 2020-10-09T17:16:07.141 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-09T17:16:07.141 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-09T17:16:07.145 INFO:tasks.rgw.c1.client.1:Started 2020-10-09T17:16:07.146 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-10-09T17:16:07.163 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:16:07.163 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:16:07.167 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8001: Connection refused 2020-10-09T17:16:07.167 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:16:08.169 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-10-09T17:16:08.184 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:16:08.184 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:16:08.189 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-09T17:16:08.190 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-10-09T17:16:08.191 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-10-09T17:16:08.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:08.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:08.192 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi060.front.sepia.ceph.com:8000 --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --url http://smithi060.front.sepia.ceph.com:8000 --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:16:08.241 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:08.241 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:08.285 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:16:08.282+0000 7fd719e13700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.24366018-2cc1-4951-82d7-d435fdbed094.control err=-107 2020-10-09T17:16:08.309 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-realm", 2020-10-09T17:16:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "current_period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:08.310 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2 2020-10-09T17:16:08.310 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:08.311 INFO:tasks.rgw_multisite:creating zone.. 2020-10-09T17:16:08.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:08.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--endpoints', 'http://smithi184.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:08.313 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --endpoints http://smithi184.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:16:08.363 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:08.364 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:08.435 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:16:08.430+0000 7f80c08780c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:16:08.436 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:16:08.430+0000 7f80c08780c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:16:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:16:08.434+0000 7f80c08780c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:16:08.437 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:16:08.434+0000 7f80c08780c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:16:08.444 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone4.rgw.control", 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-10-09T17:16:08.445 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone4.rgw.log", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-10-09T17:16:08.446 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:16:08.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:16:08.448 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:08.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_config": { 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: "event_retention_days": 7, 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: "start_with_full_sync": false 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:08.450 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-10-09T17:16:08.451 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:08.458 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-10-09T17:16:09.013 INFO:teuthology.orchestra.run.smithi184.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-10-09T17:16:09.046 INFO:teuthology.orchestra.run.smithi184:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-10-09T17:16:09.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-10-09T17:16:10.034 INFO:teuthology.orchestra.run.smithi184.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-10-09T17:16:10.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:10.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:10.060 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:16:10.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:10.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:10.125 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:10.125 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:16:10.125 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:16:10.125 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone4.rgw.meta:root", 2020-10-09T17:16:10.125 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone4.rgw.control", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone4.rgw.log", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2020-10-09T17:16:10.126 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone4.rgw.otp", 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:16:10.127 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:10.128 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2020-10-09T17:16:10.129 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_config": { 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "event_retention_days": 7, 2020-10-09T17:16:10.130 INFO:teuthology.orchestra.run.smithi184.stdout: "start_with_full_sync": false 2020-10-09T17:16:10.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:10.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "", 2020-10-09T17:16:10.131 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2020-10-09T17:16:10.131 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:10.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:10.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:10.133 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:16:10.211 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:10.211 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:18.255 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:16:18.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:18.329 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:18.329 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 4, 2020-10-09T17:16:18.329 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "b12b9dd8-e452-43dd-8255-07a03ef04ce8", 2020-10-09T17:16:18.329 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [], 2020-10-09T17:16:18.329 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:16:18.330 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:18.331 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:16:18.332 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:16:18.333 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:16:18.334 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:16:18.334 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:16:18.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.334 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:16:18.335 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.336 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:16:18.337 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:16:18.338 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:16:18.339 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:16:18.340 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.341 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:16:18.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.342 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:16:18.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:16:18.343 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:16:18.343 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:16:18.343 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:18.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.343 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.344 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:16:18.345 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:16:18.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:16:18.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:16:18.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:16:18.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:18.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:16:18.397 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:16:18.398 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:16:18.398 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:16:18.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.398 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:18.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:18.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:16:18.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:16:18.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:18.400 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:18.401 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-10-09T17:16:18.401 INFO:teuthology.orchestra.run:waiting for 300 2020-10-09T17:16:18.442 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:16:18.442+0000 7f60827fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 16309) UID: 0 2020-10-09T17:16:18.442 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:16:18.442+0000 7f609e1f1100 -1 shutting down 2020-10-09T17:16:20.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-10-09T17:16:24.503 INFO:tasks.rgw.c2.client.1:Stopped 2020-10-09T17:16:24.504 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-10-09T17:16:24.504 INFO:teuthology.orchestra.run.smithi184:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-10-09T17:16:24.507 INFO:tasks.rgw.c2.client.1:Started 2020-10-09T17:16:24.507 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001 2020-10-09T17:16:24.524 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:16:24.524 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:16:24.528 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi184.front.sepia.ceph.com port 8001: Connection refused 2020-10-09T17:16:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:16:25.530 INFO:teuthology.orchestra.run.smithi184:> curl http://smithi184.front.sepia.ceph.com:8001 2020-10-09T17:16:25.548 INFO:teuthology.orchestra.run.smithi184.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:16:25.548 INFO:teuthology.orchestra.run.smithi184.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:16:25.552 INFO:teuthology.orchestra.run.smithi184.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-10-09T17:16:25.553 INFO:teuthology.orchestra.run.smithi184.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005f809ae9-10ea-test-zone4","HostId":"10ea-test-zone4-test-zonegroup"} 2020-10-09T17:16:25.553 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-10-09T17:16:25.553 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-10-09T17:16:25.626 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-10-09T17:16:25.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:25.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:16:25.628 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:16:25.677 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:25.678 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:25.880 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:16:25.880 INFO:teuthology.orchestra.run.smithi060.stdout: "user_id": "rgw-multisite-test-user", 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "display_name": "Multisite Test User", 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "email": "", 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "suspended": 0, 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "max_buckets": 1000, 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "subusers": [], 2020-10-09T17:16:25.881 INFO:teuthology.orchestra.run.smithi060.stdout: "keys": [ 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: "user": "rgw-multisite-test-user", 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "B0DD35KPK51LGI8D73VZ", 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "RMBEhJQNEHkgDTALqZ6HtqCuJ8qv6LHC3z5vnEUF" 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:16:25.882 INFO:teuthology.orchestra.run.smithi060.stdout: "swift_keys": [], 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": [], 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "op_mask": "read, write, delete", 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "", 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "default_storage_class": "", 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_tags": [], 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:16:25.883 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:16:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "temp_url_keys": [], 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "rgw", 2020-10-09T17:16:25.885 INFO:teuthology.orchestra.run.smithi060.stdout: "mfa_ids": [] 2020-10-09T17:16:25.886 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:16:25.886 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:16:25.911 INFO:rgw_multi.tests:meta checkpoint 2020-10-09T17:16:25.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:25.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:25.912 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:25.949 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:25.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:26.161 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:16:26.194 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:16:26.194 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:16:26.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:16:26.196 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:16:26.197 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:16:26.197 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:16:26.197 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:16:26.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:26.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:26.199 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:26.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-10-09T17:16:26.238 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:26.239 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:26.500 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:26.501 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:26.501 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:26.501 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:26.501 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:26.501 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:26.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:26.503 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:26.504 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:26.506 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:26.507 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:26.508 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:26.509 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:26.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:26.510 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:26.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:31.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:31.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:31.512 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:31.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:31.551 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:31.784 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:31.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:31.786 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:31.788 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:31.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:31.790 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:31.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:31.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:31.793 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:31.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:31.794 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:31.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:36.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:36.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:36.797 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:36.837 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:36.838 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:37.088 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:37.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:37.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:42.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:42.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:42.101 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:42.140 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:42.140 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:42.392 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:42.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:42.393 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:42.393 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:42.393 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:42.393 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:42.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:42.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:42.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:42.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:42.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:42.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:42.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:42.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:42.402 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:42.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:47.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:47.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:47.405 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:47.443 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:47.443 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:47.682 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:47.682 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:47.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:47.684 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:47.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:47.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:47.687 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:47.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:47.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:47.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:47.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:47.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:47.689 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:47.690 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:47.691 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:47.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:47.692 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:47.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:52.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:52.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:52.695 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:52.734 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:52.785 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:52.990 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:52.990 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:52.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:52.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:52.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:52.994 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:52.995 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:52.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:52.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:52.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:52.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:53.000 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:53.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:16:58.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:58.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:16:58.003 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:16:58.049 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:16:58.049 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:16:58.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:16:58.268 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:16:58.268 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:16:58.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:58.269 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:58.270 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:58.271 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:58.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:58.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:16:58.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:16:58.275 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:16:58.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:16:58.277 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:16:58.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:16:58.277 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:16:58.278 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:03.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:03.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:03.280 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:03.319 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:03.319 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:03.567 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:03.568 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:03.568 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:03.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:03.569 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:03.569 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:03.569 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:03.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:03.570 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:03.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:03.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:03.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:03.571 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:03.571 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:03.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:03.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:03.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:03.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:03.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:03.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:03.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:03.574 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:03.574 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:03.574 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:03.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:03.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:03.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:03.576 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:03.576 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:03.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:03.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:03.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:03.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:03.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:03.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:03.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:03.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:03.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:03.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:03.580 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:03.580 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:03.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:03.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:03.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:03.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:03.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:03.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.582 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:03.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:03.583 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:03.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:03.583 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:03.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:03.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:03.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:08.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:08.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:08.588 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:08.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:08.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:08.878 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:08.878 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:08.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:08.880 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:08.882 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:08.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:08.884 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:08.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:08.886 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:08.887 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:08.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:08.888 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:08.889 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:13.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:13.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:13.891 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:13.929 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:13.930 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:14.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:14.240 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:14.241 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:14.242 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:14.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:14.245 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:14.246 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:14.247 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:14.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:14.249 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:14.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:19.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:19.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:19.253 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:19.296 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:19.296 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:19.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:19.536 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:19.537 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:19.539 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:19.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:19.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:19.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:19.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:19.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:19.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:19.542 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:19.542 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:19.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:19.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:19.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:19.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:19.546 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:19.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:19.546 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:19.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:19.546 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:19.547 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:24.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:24.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:24.549 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:24.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:24.590 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:24.831 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:24.832 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:24.832 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:24.832 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:24.832 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:24.833 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:24.833 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:24.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:24.833 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:24.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:24.834 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:24.834 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:24.835 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:24.835 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:24.835 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:24.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:24.836 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:24.836 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:24.836 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:24.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:24.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:24.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:24.838 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:24.838 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:24.838 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:24.838 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:24.839 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:24.839 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:24.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:24.840 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:24.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:24.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:24.842 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:24.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:24.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:24.844 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:24.844 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:24.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:24.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:24.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:24.845 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:24.846 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:24.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:24.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:24.849 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:24.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:29.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:29.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:29.852 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:29.889 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:29.890 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:30.115 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:30.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:30.115 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:30.116 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:30.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:30.117 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:30.118 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:30.119 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:30.120 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:30.121 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:30.123 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:30.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:30.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:30.125 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:30.126 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:35.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:35.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:35.128 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:35.172 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:35.172 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:35.415 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:35.415 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:35.415 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:35.416 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:35.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:35.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:35.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:35.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:35.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:35.421 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:35.421 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:35.422 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:35.422 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:35.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:35.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:35.423 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:35.424 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:35.424 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:35.425 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:35.426 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:35.427 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:35.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:35.427 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:35.428 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:40.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:40.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:40.431 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:40.472 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:40.472 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:40.727 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:40.728 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:40.728 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:40.728 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:40.728 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:40.729 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:40.730 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:40.730 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:40.730 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:40.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:40.731 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:40.731 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:40.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:40.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:40.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:40.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:40.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:40.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:40.733 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:40.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:40.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:40.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:40.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:40.734 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:40.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:40.736 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:40.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:40.738 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:40.739 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:40.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:40.739 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:40.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:45.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:45.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:45.742 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:45.785 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:45.785 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:46.007 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:46.007 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:46.008 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:46.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:46.010 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:46.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:46.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:46.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:46.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:46.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:46.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:46.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:46.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:46.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:46.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.022 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:46.023 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:46.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:46.024 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:46.026 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:51.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:51.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:51.029 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:51.069 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:51.069 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:51.312 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:51.312 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:51.312 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:51.313 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:51.314 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:51.314 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:51.314 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:51.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:51.315 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:51.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:51.316 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:51.316 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:51.317 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:51.318 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:51.318 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:51.318 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:51.319 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:51.319 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:51.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:51.320 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:51.320 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:51.320 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:51.321 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:51.321 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:51.321 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:51.321 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:51.321 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:51.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:51.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:51.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:51.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:51.323 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:51.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:51.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:51.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:51.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:51.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:51.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:51.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:51.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:51.326 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:51.326 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:51.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:51.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:51.327 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:51.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:51.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:51.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:51.329 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:51.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.329 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.329 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:51.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:51.330 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:51.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:51.330 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:51.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:51.331 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:51.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:17:56.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:56.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:17:56.334 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:17:56.375 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:17:56.375 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:17:56.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:17:56.636 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:17:56.636 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:17:56.636 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:17:56.636 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:17:56.637 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:56.638 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:56.639 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:56.641 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:17:56.642 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:17:56.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:17:56.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:17:56.645 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:17:56.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:17:56.645 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:17:56.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:18:01.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:01.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:01.649 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:01.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:01.690 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:01.917 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:01.918 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:01.919 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:01.920 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:01.921 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:01.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:01.924 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:01.924 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:01.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:01.925 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.926 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:01.927 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:01.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:18:06.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:06.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:06.931 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:06.973 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:06.973 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:07.222 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:07.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:07.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:07.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:07.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:07.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:07.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:07.229 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:07.230 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:07.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:07.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.232 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:07.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:07.232 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:07.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:07.232 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:07.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:07.233 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:07.234 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:18:12.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:12.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:12.237 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:12.277 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:12.277 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:12.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:12.537 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:12.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:12.539 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:12.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:12.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:12.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:12.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:12.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:12.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:12.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:12.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:12.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:12.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:12.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.550 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:12.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:12.550 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:12.550 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:12.550 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:12.551 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:12.551 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:12.552 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:18:17.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:17.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:17.554 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:17.597 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:17.597 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:17.841 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:17.841 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:17.841 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:17.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:17.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:17.843 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:17.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:17.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:17.846 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:17.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:17.848 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:17.850 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:17.851 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1602263785.872885_64.1 target= 2020-10-09T17:18:22.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:22.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:22.853 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:22.894 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:22.895 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:23.149 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:23.150 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:23.150 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:23.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:23.150 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:23.151 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:23.151 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.151 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:23.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:23.152 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.152 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.152 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:23.152 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.153 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:23.153 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.153 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:23.153 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.154 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:23.154 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.154 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.155 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.155 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.155 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.156 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.158 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.158 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.158 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.161 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:23.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.161 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:23.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:23.162 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:23.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:23.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:23.164 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:23.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.165 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:23.207 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:23.443 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:23.443 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:23.444 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:23.445 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:23.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:23.447 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:23.448 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:23.449 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:23.450 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:23.451 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:23.452 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:23.453 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:23.453 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:23.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:23.454 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:23.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.455 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:23.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:23.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:23.764 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:23.764 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:23.765 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.766 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.767 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.768 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.769 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:23.769 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.770 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:23.771 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:23.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:23.773 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:23.774 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:23.775 INFO:rgw_multi.tests:meta checkpoint 2020-10-09T17:18:23.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:23.776 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:23.816 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:24.022 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:24.053 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:24.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.054 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:24.054 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:24.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.055 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:24.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.055 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.056 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:24.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:24.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.058 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:24.098 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:24.099 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:24.323 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:24.323 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:24.324 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:24.324 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:24.324 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:24.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.327 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:24.327 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.328 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:24.330 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.331 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.332 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.336 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:24.338 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:24.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.339 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:24.341 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:24.341 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:24.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.342 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:24.383 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:24.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:24.625 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.626 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:24.627 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:24.629 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.630 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:24.631 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:24.632 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:24.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:24.634 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:24.635 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:24.636 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:24.636 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:24.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:24.637 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:24.679 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:24.679 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:24.946 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:24.947 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:24.948 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:24.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.948 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:24.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:24.949 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.949 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.949 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:24.950 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.952 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.953 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:24.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.954 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.954 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:24.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:24.956 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:24.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:24.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:24.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:24.958 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:24.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:24.959 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:24.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:24.962 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-10-09T17:18:24.971 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-1 2020-10-09T17:18:28.746 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-2 2020-10-09T17:18:32.101 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-3 2020-10-09T17:18:32.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.151 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:32.183 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:32.184 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:32.368 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:32.397 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:32.397 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.397 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263912.133520_63.1", 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:32.133520Z" 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:32.399 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:32.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.400 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:32.401 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:32.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.402 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:32.441 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:32.441 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:32.680 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:32.680 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:32.680 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:32.681 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:32.682 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:32.683 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263912.133520_63.1", 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.133520Z", 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:32.689 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:32.690 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:32.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:32.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.692 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:32.752 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:32.753 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:32.973 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:32.974 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:32.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.975 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:32.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.975 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:32.976 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:32.977 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:32.978 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263912.133520_63.1", 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.133520Z", 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:32.981 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:32.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:32.983 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:32.983 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:32.983 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:32.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:32.983 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:32.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:32.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:32.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:32.986 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:33.027 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:33.028 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:33.285 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263912.133520_63.1", 2020-10-09T17:18:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.133520Z", 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.294 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:33.295 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:33.341 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-10-09T17:18:33.344 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-4 2020-10-09T17:18:33.370 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-5 2020-10-09T17:18:33.422 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-6 2020-10-09T17:18:33.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:33.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:33.469 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:33.512 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:33.512 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:33.690 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:33.723 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:33.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:33.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:33.726 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:33.450323Z" 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:33.727 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:33.729 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:33.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:33.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:33.731 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:33.773 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:33.774 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:33.989 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:33.999 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:34.000 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:34.001 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:34.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.002 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:34.045 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:34.045 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:34.288 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:34.289 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:34.289 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:34.289 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:34.289 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:34.290 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:34.292 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:34.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:34.294 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:34.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:34.296 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.297 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:34.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:34.299 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:34.300 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:34.300 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:34.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:34.300 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:34.301 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:34.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:34.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.303 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:34.343 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:34.343 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:34.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:34.609 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:34.610 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:34.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:34.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:34.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:34.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:34.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:34.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:34.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:34.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:34.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:34.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:34.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:34.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:34.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:34.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:34.620 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:34.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:34.622 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:34.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:34.623 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:34.624 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:34.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:34.959 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:34.994 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:34.995 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:35.203 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:35.238 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:33.450323Z" 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.240 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:35.241 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:35.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.241 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:35.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:35.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.243 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:35.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:35.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:35.529 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:35.529 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:35.529 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:35.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:35.531 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:35.532 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:35.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:35.534 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:35.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:35.536 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:35.537 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:35.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:35.538 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:35.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:35.538 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:35.539 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:35.539 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:35.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.541 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:35.581 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:35.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:35.818 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:35.819 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:35.819 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:35.820 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:35.820 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:35.820 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:35.820 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:35.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.821 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:35.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.821 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:35.822 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:35.823 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:35.823 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:35.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.824 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.824 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:35.824 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:35.825 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:35.825 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:35.825 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:35.826 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:35.826 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:35.826 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:35.827 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:35.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.828 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:35.828 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:35.828 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:35.829 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:35.829 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:35.829 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:35.831 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:35.831 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:35.831 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:35.832 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:35.832 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:35.832 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:35.833 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:35.833 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:35.833 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:35.833 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.834 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:35.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:35.835 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:35.835 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:35.835 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:35.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:35.836 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:35.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:35.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:35.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:35.840 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:35.880 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:35.880 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:36.146 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263913.450323_76.1", 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323Z", 2020-10-09T17:18:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.230 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:36.230 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.232 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:36.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.232 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:36.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:36.233 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:36.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.233 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:36.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:36.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-10-09T17:18:36.279 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-7 2020-10-09T17:18:36.302 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-8 2020-10-09T17:18:36.358 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-9 2020-10-09T17:18:36.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.403 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:36.438 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:36.633 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:36.666 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:36.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:36.667 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:36.667 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:36.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:36.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:36.668 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:36.668 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:36.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:36.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:36.669 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263916.384719_90.1", 2020-10-09T17:18:36.669 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:36.384719Z" 2020-10-09T17:18:36.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:36.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:36.670 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:36.670 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:36.670 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:36.670 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:36.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:36.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.674 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:36.717 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:36.746 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:36.964 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:36.964 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.965 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.966 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.967 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.968 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263916.384719_90.1", 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.384719Z", 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:36.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:36.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:36.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:36.973 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:36.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:36.975 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:36.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:36.976 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:37.018 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:37.019 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:37.263 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:37.263 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:37.264 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:37.265 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:37.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:37.267 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:37.268 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:37.269 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:37.269 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263916.384719_90.1", 2020-10-09T17:18:37.269 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:37.269 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:37.269 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.384719Z", 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:37.270 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:37.271 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:37.272 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:37.273 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:37.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:37.273 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:37.274 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:37.274 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:37.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:37.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:37.276 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:37.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:37.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:37.580 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263916.384719_90.1", 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.384719Z", 2020-10-09T17:18:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:37.591 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:37.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:37.592 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:37.593 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:37.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:37.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:37.915 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:37.951 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:37.952 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:38.154 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:38.188 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:38.202 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.203 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:38.203 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:38.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.204 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:38.204 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:38.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.205 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:38.206 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:38.206 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.206 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:38.207 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:38.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.209 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:38.251 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:38.251 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:38.482 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:38.483 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:38.483 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:38.483 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:38.483 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:38.483 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:38.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:38.485 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:38.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:38.485 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:38.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:38.486 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:38.487 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:38.488 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:38.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:38.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:38.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:38.489 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:38.490 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:38.491 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:38.492 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:38.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:38.493 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:38.494 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:38.494 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:38.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.496 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:38.539 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:38.568 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:38.767 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:38.767 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:38.767 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:38.767 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.768 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:38.770 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:38.771 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:38.772 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:38.773 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:38.774 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:38.774 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:38.774 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.776 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:38.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:38.776 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:38.777 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:38.777 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:38.777 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:38.777 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:38.779 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:38.779 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:38.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:38.780 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:38.821 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:38.821 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:39.086 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:39.086 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:39.087 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:39.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:39.087 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:39.088 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:39.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:39.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:39.089 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:39.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:39.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:39.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:39.096 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:39.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:39.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:39.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:39.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:39.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:39.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:39.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:39.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:39.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:39.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:39.100 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:39.101 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:39.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:39.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:39.103 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:39.113 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:39.114 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:39.115 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:39.116 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:39.117 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:39.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:39.124 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:39.128 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:39.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-10-09T17:18:39.172 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-10 2020-10-09T17:18:39.205 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-11 2020-10-09T17:18:39.265 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-12 2020-10-09T17:18:39.343 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.340+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: bhVdw7AtKI+7FKPw{1}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.343 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.340+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: bhVdw7AtKI+7FKPw{1}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.344 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.340+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: bhVdw7AtKI+7FKPw{1}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.344 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.340+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: bhVdw7AtKI+7FKPw{1}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.344 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.340+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.345 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.344+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: bhVdw7AtKI+7FKPw{2}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=uu4NpuTcvV8hGASC}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: uu4NpuTcvV8hGASC,0=op: Op::set_head, part_num: 0, part_tag: uu4NpuTcvV8hGASC}} 2020-10-09T17:18:39.345 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.344+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: uu4NpuTcvV8hGASC 2020-10-09T17:18:39.354 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.348+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: uu4NpuTcvV8hGASC 2020-10-09T17:18:39.354 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.348+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.354 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.348+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.354 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.352+0000 7f3b157fa700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: bhVdw7AtKI+7FKPw{3}, oid_prefix: data_log.3.74nyUmfNlhN, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: uu4NpuTcvV8hGASC, tags: {{0=uu4NpuTcvV8hGASC}}, journal: {{{}} 2020-10-09T17:18:39.389 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.388+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.1, version: VWLeAjkEExzG1V+d{1}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.390 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.388+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.1, version: VWLeAjkEExzG1V+d{1}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.390 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.388+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.1, version: VWLeAjkEExzG1V+d{1}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.390 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.388+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.1, version: VWLeAjkEExzG1V+d{1}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.390 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.388+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.392 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.392+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.1, version: VWLeAjkEExzG1V+d{2}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=F29pRpyDI28a9k59}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: F29pRpyDI28a9k59,0=op: Op::set_head, part_num: 0, part_tag: F29pRpyDI28a9k59}} 2020-10-09T17:18:39.392 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.392+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: F29pRpyDI28a9k59 2020-10-09T17:18:39.395 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.392+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: F29pRpyDI28a9k59 2020-10-09T17:18:39.395 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.392+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.396 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.392+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.398 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.396+0000 7f3b017e2700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.1, version: VWLeAjkEExzG1V+d{3}, oid_prefix: data_log.1.LA/iUW9e4dg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: F29pRpyDI28a9k59, tags: {{0=F29pRpyDI28a9k59}}, journal: {{{}} 2020-10-09T17:18:39.450 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.448+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: tNSM8PD9Zz6cUaLe{1}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.450 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.448+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: tNSM8PD9Zz6cUaLe{1}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.451 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.448+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: tNSM8PD9Zz6cUaLe{1}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.451 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.448+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: tNSM8PD9Zz6cUaLe{1}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.451 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.448+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.458 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.452+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: tNSM8PD9Zz6cUaLe{2}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=76xyAwCxbwjOLOBO}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 76xyAwCxbwjOLOBO,0=op: Op::set_head, part_num: 0, part_tag: 76xyAwCxbwjOLOBO}} 2020-10-09T17:18:39.459 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.452+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: 76xyAwCxbwjOLOBO 2020-10-09T17:18:39.460 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.452+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: 76xyAwCxbwjOLOBO 2020-10-09T17:18:39.460 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.452+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.460 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.452+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.460 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:18:39.456+0000 7f3af27c4700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: tNSM8PD9Zz6cUaLe{3}, oid_prefix: data_log.2.bBeN+FLJWop, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: 76xyAwCxbwjOLOBO, tags: {{0=76xyAwCxbwjOLOBO}}, journal: {{{}} 2020-10-09T17:18:39.573 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.574+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: zCyf9qq+1u2xR99H{1}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.573 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.574+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: zCyf9qq+1u2xR99H{1}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.574 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.574+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: zCyf9qq+1u2xR99H{1}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.574 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.574+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: zCyf9qq+1u2xR99H{1}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.574 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.574+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.577 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.578+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: zCyf9qq+1u2xR99H{2}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=X9r6AOHwSiJ8lTAs}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: X9r6AOHwSiJ8lTAs,0=op: Op::set_head, part_num: 0, part_tag: X9r6AOHwSiJ8lTAs}} 2020-10-09T17:18:39.577 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.578+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: X9r6AOHwSiJ8lTAs 2020-10-09T17:18:39.580 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.582+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: X9r6AOHwSiJ8lTAs 2020-10-09T17:18:39.581 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.582+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.581 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.582+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.583 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.582+0000 7fd75e69c700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: zCyf9qq+1u2xR99H{3}, oid_prefix: data_log.3.PfFOjc/IWBR, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: X9r6AOHwSiJ8lTAs, tags: {{0=X9r6AOHwSiJ8lTAs}}, journal: {{{}} 2020-10-09T17:18:39.624 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.1, version: i/v1oW/gA9yR0Hf7{1}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.625 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.1, version: i/v1oW/gA9yR0Hf7{1}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.625 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.1, version: i/v1oW/gA9yR0Hf7{1}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.625 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.1, version: i/v1oW/gA9yR0Hf7{1}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.626 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.627 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.1, version: i/v1oW/gA9yR0Hf7{2}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=WG8C15Cc0Q5ne7PE}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: WG8C15Cc0Q5ne7PE,0=op: Op::set_head, part_num: 0, part_tag: WG8C15Cc0Q5ne7PE}} 2020-10-09T17:18:39.627 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.626+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: WG8C15Cc0Q5ne7PE 2020-10-09T17:18:39.629 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.630+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: WG8C15Cc0Q5ne7PE 2020-10-09T17:18:39.630 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.630+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.630 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.630+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.631 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.630+0000 7fd750e81700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.1, version: i/v1oW/gA9yR0Hf7{3}, oid_prefix: data_log.1.8/Nh9GjYM0Z, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: WG8C15Cc0Q5ne7PE, tags: {{0=WG8C15Cc0Q5ne7PE}}, journal: {{{}} 2020-10-09T17:18:39.658 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.652+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: OdU5XMvOI/jg8zS9{1}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.659 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.652+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: OdU5XMvOI/jg8zS9{1}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.659 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.652+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: OdU5XMvOI/jg8zS9{1}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.659 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.652+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: OdU5XMvOI/jg8zS9{1}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.660 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.652+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.676 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.660+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: OdU5XMvOI/jg8zS9{2}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=AouZyZCsxRlROjKi}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: AouZyZCsxRlROjKi,0=op: Op::set_head, part_num: 0, part_tag: AouZyZCsxRlROjKi}} 2020-10-09T17:18:39.676 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.660+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: AouZyZCsxRlROjKi 2020-10-09T17:18:39.676 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.668+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: AouZyZCsxRlROjKi 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.668+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.668+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.668+0000 7f40a8ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: OdU5XMvOI/jg8zS9{3}, oid_prefix: data_log.3.PhglkOPoiz1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: AouZyZCsxRlROjKi, tags: {{0=AouZyZCsxRlROjKi}}, journal: {{{}} 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: 3z7sXqIbnKcHd7QL{1}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: 3z7sXqIbnKcHd7QL{1}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.677 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: 3z7sXqIbnKcHd7QL{1}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: 3z7sXqIbnKcHd7QL{1}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.1, version: USMFU4Nua+ZMFD2I{1}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.1, version: USMFU4Nua+ZMFD2I{1}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.1, version: USMFU4Nua+ZMFD2I{1}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.678 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.1, version: USMFU4Nua+ZMFD2I{1}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.679 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.672+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.679 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: 3z7sXqIbnKcHd7QL{2}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=TjpsysT9OI3bY7aF}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: TjpsysT9OI3bY7aF,0=op: Op::set_head, part_num: 0, part_tag: TjpsysT9OI3bY7aF}} 2020-10-09T17:18:39.679 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: TjpsysT9OI3bY7aF 2020-10-09T17:18:39.681 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.666+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: XSL0ah9LAwm7POXm{1}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.666+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: XSL0ah9LAwm7POXm{1}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.666+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: XSL0ah9LAwm7POXm{1}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.666+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: XSL0ah9LAwm7POXm{1}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.666+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.670+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: XSL0ah9LAwm7POXm{2}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=uCU3UsAPuAB99l3Z}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: uCU3UsAPuAB99l3Z,0=op: Op::set_head, part_num: 0, part_tag: uCU3UsAPuAB99l3Z}} 2020-10-09T17:18:39.682 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.670+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: uCU3UsAPuAB99l3Z 2020-10-09T17:18:39.683 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.674+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: uCU3UsAPuAB99l3Z 2020-10-09T17:18:39.683 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.674+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.683 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.674+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.683 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:18:39.678+0000 7fd91a7f4700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: XSL0ah9LAwm7POXm{3}, oid_prefix: data_log.2.KEgxCJYxQjD, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: uCU3UsAPuAB99l3Z, tags: {{0=uCU3UsAPuAB99l3Z}}, journal: {{{}} 2020-10-09T17:18:39.683 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.1, version: USMFU4Nua+ZMFD2I{2}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=mAuGlpYVsT5jukaV}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: mAuGlpYVsT5jukaV,0=op: Op::set_head, part_num: 0, part_tag: mAuGlpYVsT5jukaV}} 2020-10-09T17:18:39.683 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: mAuGlpYVsT5jukaV 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: TjpsysT9OI3bY7aF 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: mAuGlpYVsT5jukaV 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:18:39.684 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.676+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:18:39.685 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.680+0000 7f40abfff700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: 3z7sXqIbnKcHd7QL{3}, oid_prefix: data_log.2.CSAj2WDgH+1, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: TjpsysT9OI3bY7aF, tags: {{0=TjpsysT9OI3bY7aF}}, journal: {{{}} 2020-10-09T17:18:39.685 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:18:39.680+0000 7f409e7fc700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.1, version: USMFU4Nua+ZMFD2I{3}, oid_prefix: data_log.1.szQWweS2FIg, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: mAuGlpYVsT5jukaV, tags: {{0=mAuGlpYVsT5jukaV}}, journal: {{{}} 2020-10-09T17:18:40.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.005 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:40.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:40.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:40.222 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:40.252 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:40.252 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:40.253 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263919.301783_71.1", 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:39.301783Z" 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.254 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:40.255 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:40.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.257 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:40.300 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:40.301 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:40.533 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:40.534 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:40.534 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:40.534 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:40.534 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:40.535 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:40.536 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:40.537 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:40.538 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:40.538 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:40.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:40.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:40.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:40.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:40.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263919.301783_71.1", 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.301783Z", 2020-10-09T17:18:40.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:40.543 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:40.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:40.544 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:40.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:40.545 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:40.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.547 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:40.590 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:40.591 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:40.811 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:40.811 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:40.811 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:40.811 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:40.812 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:40.812 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:40.812 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:40.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.812 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:40.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.813 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:40.813 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:40.813 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:40.813 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:40.814 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:40.814 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:40.814 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:40.814 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:40.814 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:40.815 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:40.816 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:40.816 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:40.816 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:40.816 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:40.816 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:40.817 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:40.817 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:40.817 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:40.819 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:40.820 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263919.301783_71.1", 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:40.821 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.301783Z", 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:40.822 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:40.823 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:40.823 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:40.823 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:40.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:40.824 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:40.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:40.826 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:40.868 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:40.869 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:41.134 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:41.134 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:41.134 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:41.134 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:41.135 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:41.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:41.137 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:41.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:41.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:41.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:41.141 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:41.141 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:41.141 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263919.301783_71.1", 2020-10-09T17:18:41.141 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:41.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:41.142 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:41.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.301783Z", 2020-10-09T17:18:41.142 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:41.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.143 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:41.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:41.143 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:41.144 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:41.144 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:41.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:41.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:41.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:41.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:41.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:41.146 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:41.231 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:41.232 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:41.467 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:41.466+0000 7fd4f3c330c0 1 bucket sync caught up with source: 2020-10-09T17:18:41.467 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:41.467 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:41.468 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:41.466+0000 7fd4f3c330c0 0 bucket checkpoint complete 2020-10-09T17:18:41.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-10 zones={test-zone2, test-zone1} 2020-10-09T17:18:41.510 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:41.529 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:41.530 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:41.530 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:41.531 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:41.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:41.540 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:41.541 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:41.541 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:41.542 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:41.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:41.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:41.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:41.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:41.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:42.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:42.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:42.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:42.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:42.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:42.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:42.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:42.687 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-10 zones={test-zone2, test-zone1} 2020-10-09T17:18:42.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:42.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:42.689 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:42.737 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:42.738 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:42.952 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:42.952+0000 7f1ad80500c0 1 bucket sync caught up with source: 2020-10-09T17:18:42.953 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:42.953 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:42.953 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:42.952+0000 7f1ad80500c0 0 bucket checkpoint complete 2020-10-09T17:18:42.978 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-10 zones={test-zone3, test-zone1} 2020-10-09T17:18:42.987 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:42.995 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:42.995 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:42.996 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:42.996 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:42.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:43.002 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:43.003 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:43.003 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:43.004 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:43.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:43.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:43.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:43.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:43.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:43.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:43.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:43.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:43.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:43.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:43.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:44.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:44.118 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-10 zones={test-zone3, test-zone1} 2020-10-09T17:18:44.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:44.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:44.119 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:44.170 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:44.171 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:44.403 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:44.402+0000 7ff0e96d10c0 1 bucket sync caught up with source: 2020-10-09T17:18:44.403 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:44.403 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.11.5, , 00000000001.4.5, , , , 00000000001.6.5, 00000000001.4.5, , ] 2020-10-09T17:18:44.404 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:44.402+0000 7ff0e96d10c0 0 bucket checkpoint complete 2020-10-09T17:18:44.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:44.537 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:44.589 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:44.589 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:44.799 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:44.796+0000 7fe5685620c0 1 bucket sync caught up with source: 2020-10-09T17:18:44.799 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.6.5, , 00000000001.3.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2020-10-09T17:18:44.799 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.6.5, , 00000000001.3.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2020-10-09T17:18:44.799 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:44.796+0000 7fe5685620c0 0 bucket checkpoint complete 2020-10-09T17:18:44.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-11 zones={test-zone1, test-zone2} 2020-10-09T17:18:44.839 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:44.845 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:44.846 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:44.846 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:44.847 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:44.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:44.853 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:44.854 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:44.854 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:44.854 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:44.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:44.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:45.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:45.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:45.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:45.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:45.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:45.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:45.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:45.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:45.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:45.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:45.974 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-11 zones={test-zone1, test-zone2} 2020-10-09T17:18:45.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:45.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:45.975 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:46.025 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:46.026 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:46.258 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:46.256+0000 7fcabbaac0c0 1 bucket sync caught up with source: 2020-10-09T17:18:46.258 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.6.5, , 00000000001.3.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2020-10-09T17:18:46.258 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.6.5, , 00000000001.3.5, , , , 00000000001.7.5, 00000000001.9.5, , ] 2020-10-09T17:18:46.258 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:46.256+0000 7fcabbaac0c0 0 bucket checkpoint complete 2020-10-09T17:18:46.288 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-11 zones={test-zone3, test-zone2} 2020-10-09T17:18:46.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:46.308 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:46.309 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:46.309 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:46.310 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:46.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:46.317 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:46.317 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:46.318 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:46.319 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:46.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:46.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:46.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:46.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:46.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:46.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:46.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:46.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:47.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:47.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:47.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:47.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:47.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-11 zones={test-zone3, test-zone2} 2020-10-09T17:18:47.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:47.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:47.463 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:47.512 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:47.513 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:47.740 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:47.740+0000 7ff6211490c0 1 bucket sync caught up with source: 2020-10-09T17:18:47.741 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.11.5, , 00000000001.9.5, , , , 00000000001.8.5, 00000000001.6.5, , ] 2020-10-09T17:18:47.741 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.11.5, , 00000000001.9.5, , , , 00000000001.8.5, 00000000001.6.5, , ] 2020-10-09T17:18:47.741 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:47.740+0000 7ff6211490c0 0 bucket checkpoint complete 2020-10-09T17:18:47.767 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-12 zones={test-zone1, test-zone3} 2020-10-09T17:18:47.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:47.786 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:47.787 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:47.787 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:47.787 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:47.788 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:47.792 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:47.793 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:47.793 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:47.793 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:47.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:47.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:47.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:48.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:48.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:48.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:48.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:48.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:48.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:48.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:48.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:48.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:48.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-12 zones={test-zone1, test-zone3} 2020-10-09T17:18:48.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:48.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:48.915 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:48.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:48.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:49.190 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:49.190+0000 7f33025970c0 1 bucket sync caught up with source: 2020-10-09T17:18:49.190 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.11.5, , 00000000001.9.5, , , , 00000000001.8.5, 00000000001.6.5, , ] 2020-10-09T17:18:49.190 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.11.5, , 00000000001.9.5, , , , 00000000001.8.5, 00000000001.6.5, , ] 2020-10-09T17:18:49.190 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:49.190+0000 7f33025970c0 0 bucket checkpoint complete 2020-10-09T17:18:49.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-12 zones={test-zone2, test-zone3} 2020-10-09T17:18:49.231 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:49.239 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:49.239 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:49.239 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:49.240 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:49.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:49.246 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-10-09T17:18:49.247 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-10-09T17:18:49.248 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-10-09T17:18:49.248 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:49.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:49.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:49.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-10-09T17:18:49.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:49.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:49.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-10-09T17:18:49.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:49.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:50.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-10-09T17:18:50.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:50.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:50.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:50.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-12 zones={test-zone2, test-zone3} 2020-10-09T17:18:50.379 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-10-09T17:18:50.381 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-13 2020-10-09T17:18:50.407 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-14 2020-10-09T17:18:50.464 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-15 2020-10-09T17:18:50.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:50.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:50.680 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:50.725 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:50.725 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:50.891 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:18:50.915 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:18:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:50.915 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:50.916 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:18:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:50.917 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263930.495381_95.1", 2020-10-09T17:18:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:50.495381Z" 2020-10-09T17:18:50.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:50.919 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:18:50.920 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:18:50.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:50.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:50.921 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:50.962 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:50.962 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:51.179 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:51.179 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:51.179 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:51.179 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:51.179 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.180 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.182 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:51.184 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263930.495381_95.1", 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.495381Z", 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:51.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.188 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:51.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:51.188 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:51.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.188 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:51.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:18:51.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:18:51.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:51.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:51.191 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:51.230 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:51.230 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:51.466 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:51.467 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:51.468 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:51.470 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:51.471 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:51.472 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:18:51.473 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263930.495381_95.1", 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.495381Z", 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.474 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:18:51.475 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:18:51.476 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:18:51.477 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:18:51.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:51.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:18:51.478 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:18:51.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:51.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:51.787 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:18:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:18:51.787 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.788 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:18:51.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:18:51.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:18:51.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263930.495381_95.1", 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.495381Z", 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:18:51.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:18:51.796 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:18:51.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:18:51.797 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:18:51.798 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:18:51.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:51.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:51.800 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:51.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:51.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:52.111 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:52.114+0000 7fa59493b0c0 1 bucket sync caught up with source: 2020-10-09T17:18:52.112 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:52.112 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:52.112 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:52.114+0000 7fa59493b0c0 0 bucket checkpoint complete 2020-10-09T17:18:52.142 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-13 zones={test-zone2, test-zone1} 2020-10-09T17:18:52.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:52.202 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:52.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:52.211 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:52.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:52.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:52.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:52.541 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-13 zones={test-zone2, test-zone1} 2020-10-09T17:18:52.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:52.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:52.543 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:52.587 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:52.587 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:52.803 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:52.804+0000 7fb73ae300c0 1 bucket sync caught up with source: 2020-10-09T17:18:52.804 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:52.804 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:52.804 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:52.804+0000 7fb73ae300c0 0 bucket checkpoint complete 2020-10-09T17:18:52.829 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-13 zones={test-zone3, test-zone1} 2020-10-09T17:18:52.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:52.891 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:52.892 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:52.900 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:52.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:53.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:53.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:53.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-13 zones={test-zone3, test-zone1} 2020-10-09T17:18:53.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:53.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:53.230 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:53.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:53.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:53.533 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:53.534+0000 7f60deb280c0 1 bucket sync caught up with source: 2020-10-09T17:18:53.533 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:53.534 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:53.534 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:53.534+0000 7f60deb280c0 0 bucket checkpoint complete 2020-10-09T17:18:53.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:53.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:53.657 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:53.701 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:53.702 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:53.942 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:53.940+0000 7faf4eea80c0 1 bucket sync caught up with source: 2020-10-09T17:18:53.943 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:18:53.943 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:18:53.943 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:53.940+0000 7faf4eea80c0 0 bucket checkpoint complete 2020-10-09T17:18:53.974 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-14 zones={test-zone1, test-zone2} 2020-10-09T17:18:53.985 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:54.034 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:54.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:54.042 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:54.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:54.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:54.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:54.374 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-14 zones={test-zone1, test-zone2} 2020-10-09T17:18:54.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:54.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:54.377 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:54.425 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:54.660+0000 7f31181b00c0 1 bucket sync caught up with source: 2020-10-09T17:18:54.664 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:18:54.664 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:18:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:54.660+0000 7f31181b00c0 0 bucket checkpoint complete 2020-10-09T17:18:54.695 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-14 zones={test-zone3, test-zone2} 2020-10-09T17:18:54.705 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:54.754 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:54.755 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:54.764 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:54.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:54.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:54.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:55.094 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-14 zones={test-zone3, test-zone2} 2020-10-09T17:18:55.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:55.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:55.096 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:55.146 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:55.147 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:55.383 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:55.380+0000 7f27637310c0 1 bucket sync caught up with source: 2020-10-09T17:18:55.383 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-09T17:18:55.383 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-10-09T17:18:55.383 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:18:55.380+0000 7f27637310c0 0 bucket checkpoint complete 2020-10-09T17:18:55.408 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-15 zones={test-zone1, test-zone3} 2020-10-09T17:18:55.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:55.466 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:55.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:55.474 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:55.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:55.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:55.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:55.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-15 zones={test-zone1, test-zone3} 2020-10-09T17:18:55.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:55.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:55.811 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:55.859 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:55.860 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:56.078 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:56.078+0000 7f392f19a0c0 1 bucket sync caught up with source: 2020-10-09T17:18:56.079 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.9.5, , , , , , , , , ] 2020-10-09T17:18:56.079 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000001.9.5, , , , , , , , , ] 2020-10-09T17:18:56.079 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:56.078+0000 7f392f19a0c0 0 bucket checkpoint complete 2020-10-09T17:18:56.109 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-15 zones={test-zone2, test-zone3} 2020-10-09T17:18:56.119 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:18:56.166 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:56.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:18:56.175 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-10-09T17:18:56.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:56.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:56.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-10-09T17:18:56.507 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-15 zones={test-zone2, test-zone3} 2020-10-09T17:18:56.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:56.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:18:56.522 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:18:56.571 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:18:56.571 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:18:56.800 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:18:56.802+0000 7fd516a6a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:18:56.800 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:18:56.801 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:01.804 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:01.806+0000 7fd516a6a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:01.804 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:19:01.805 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:06.808 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:06.810+0000 7fd516a6a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:06.809 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:19:06.809 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:11.812 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:11.814+0000 7fd516a6a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:11.812 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000001.10.5, , , , , , , , , ] 2020-10-09T17:19:11.813 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:16.815 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:16.818+0000 7fd516a6a0c0 1 bucket sync caught up with source: 2020-10-09T17:19:16.815 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:16.816 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:16.816 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:16.818+0000 7fd516a6a0c0 0 bucket checkpoint complete 2020-10-09T17:19:16.842 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-13 zones={test-zone2, test-zone1} 2020-10-09T17:19:16.854 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:16.902 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:16.966 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-13 zones={test-zone2, test-zone1} 2020-10-09T17:19:16.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:16.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:16.968 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:17.014 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:17.245 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:17.244+0000 7fa9633a10c0 1 bucket sync caught up with source: 2020-10-09T17:19:17.245 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:17.245 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:17.246 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:17.244+0000 7fa9633a10c0 0 bucket checkpoint complete 2020-10-09T17:19:17.271 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-13 zones={test-zone3, test-zone1} 2020-10-09T17:19:17.280 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:17.326 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:17.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-13 zones={test-zone3, test-zone1} 2020-10-09T17:19:17.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:17.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:17.395 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:17.441 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:17.442 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:17.669 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:17.670+0000 7faac5c830c0 1 bucket sync caught up with source: 2020-10-09T17:19:17.670 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:17.670 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.12.5, , , , , , , , , ] 2020-10-09T17:19:17.670 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:17.670+0000 7faac5c830c0 0 bucket checkpoint complete 2020-10-09T17:19:17.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:17.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:17.728 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:17.776 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:18.008+0000 7f72b9b090c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:18.009 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:19:18.009 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:23.013 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:23.012+0000 7f72b9b090c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:23.013 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:19:23.014 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:28.016+0000 7f72b9b090c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:28.018 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:19:28.018 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:33.021 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:33.020+0000 7f72b9b090c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:33.022 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000001.8.5, , , , , , , , , ] 2020-10-09T17:19:33.022 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:38.026 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.024+0000 7f72b9b090c0 1 bucket sync caught up with source: 2020-10-09T17:19:38.026 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:38.027 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:38.027 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.024+0000 7f72b9b090c0 0 bucket checkpoint complete 2020-10-09T17:19:38.059 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-14 zones={test-zone1, test-zone2} 2020-10-09T17:19:38.069 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:38.122 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:38.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-14 zones={test-zone1, test-zone2} 2020-10-09T17:19:38.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:38.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:38.207 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:38.253 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:38.254 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:38.480 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.480+0000 7ffad22c40c0 1 bucket sync caught up with source: 2020-10-09T17:19:38.481 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:38.481 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.10.5, , , , , , , , , ] 2020-10-09T17:19:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.480+0000 7ffad22c40c0 0 bucket checkpoint complete 2020-10-09T17:19:38.511 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-14 zones={test-zone3, test-zone2} 2020-10-09T17:19:38.523 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:38.570 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:38.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-14 zones={test-zone3, test-zone2} 2020-10-09T17:19:38.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:38.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:38.682 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:38.726 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:38.727 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:38.922 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.920+0000 7fe417d3b0c0 1 bucket sync caught up with source: 2020-10-09T17:19:38.959 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2020-10-09T17:19:38.960 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-10-09T17:19:38.960 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:38.920+0000 7fe417d3b0c0 0 bucket checkpoint complete 2020-10-09T17:19:38.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-15 zones={test-zone1, test-zone3} 2020-10-09T17:19:38.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:39.023 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:39.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-15 zones={test-zone1, test-zone3} 2020-10-09T17:19:39.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:39.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:39.087 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:39.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:39.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:39.365 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:39.366+0000 7ff5a86820c0 1 bucket sync caught up with source: 2020-10-09T17:19:39.365 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000002.11.5, , , , , , , , , ] 2020-10-09T17:19:39.365 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000002.11.5, , , , , , , , , ] 2020-10-09T17:19:39.365 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:39.366+0000 7ff5a86820c0 0 bucket checkpoint complete 2020-10-09T17:19:39.395 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-15 zones={test-zone2, test-zone3} 2020-10-09T17:19:39.406 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:19:39.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:19:39.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-15 zones={test-zone2, test-zone3} 2020-10-09T17:19:39.516 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-10-09T17:19:39.517 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-16 2020-10-09T17:19:39.541 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-17 2020-10-09T17:19:39.597 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-18 2020-10-09T17:19:39.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:39.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:39.719 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:19:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:39.771 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:39.960 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:19:39.993 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:19:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:19:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:19:39.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:19:39.994 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263979.681384_123.1", 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:19:39.681384Z" 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:39.995 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:19:39.996 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:19:39.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:39.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:39.997 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:19:40.038 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:40.039 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:40.268 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:19:40.316 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:19:40.316 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:19:40.316 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.317 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:19:40.318 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.318 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.318 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.320 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.321 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263979.681384_123.1", 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.681384Z", 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.325 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:19:40.326 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:19:40.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:19:40.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:40.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:40.329 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:19:40.370 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:40.371 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:40.613 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:19:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:19:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:19:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:19:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:19:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:19:40.615 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:19:40.616 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:19:40.617 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:40.618 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:19:40.619 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:19:40.620 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263979.681384_123.1", 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.681384Z", 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:19:40.621 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:19:40.622 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:19:40.623 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:19:40.624 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:19:40.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:19:40.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:40.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:19:40.625 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:19:40.666 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:40.667 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:40.934 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:19:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:19:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:19:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:19:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:19:40.949 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:19:40.952 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.954 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:19:40.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263979.681384_123.1", 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.681384Z", 2020-10-09T17:19:40.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:19:40.957 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:19:40.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:19:40.958 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:19:40.959 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:19:41.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:41.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:41.015 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:41.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:41.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:41.307 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:41.310+0000 7faa9745a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:41.308 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-10-09T17:19:41.308 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:46.311 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:46.314+0000 7faa9745a0c0 1 bucket sync caught up with source: 2020-10-09T17:19:46.312 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:46.312 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:46.312 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:19:46.314+0000 7faa9745a0c0 0 bucket checkpoint complete 2020-10-09T17:19:46.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:46.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:19:46.343 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:19:46.392 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:19:46.392 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:19:46.618 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:46.616+0000 7f3ace8260c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:46.619 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:46.619 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-10-09T17:19:51.622 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:51.620+0000 7f3ace8260c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:51.623 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:51.623 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-10-09T17:19:56.626 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:19:56.624+0000 7f3ace8260c0 1 waiting for incremental sync to catch up: 2020-10-09T17:19:56.626 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.8.10, , , , , ] 2020-10-09T17:19:56.627 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-10-09T17:20:01.630 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:01.628+0000 7f3ace8260c0 1 bucket sync caught up with source: 2020-10-09T17:20:01.631 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2020-10-09T17:20:01.631 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.14.9, , , , , ] 2020-10-09T17:20:01.631 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:01.628+0000 7f3ace8260c0 0 bucket checkpoint complete 2020-10-09T17:20:01.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:01.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:01.664 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:20:01.712 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:20:01.712 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:20:01.935 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:20:01.938+0000 7f11aa8bd0c0 1 bucket sync caught up with source: 2020-10-09T17:20:01.936 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.16.9, , , , , ] 2020-10-09T17:20:01.936 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.16.9, , , , , ] 2020-10-09T17:20:01.936 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:20:01.938+0000 7f11aa8bd0c0 0 bucket checkpoint complete 2020-10-09T17:20:02.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:02.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:02.032 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:20:02.081 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:20:02.081 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:20:02.280 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:02.280+0000 7fa344ead0c0 1 bucket sync caught up with source: 2020-10-09T17:20:02.281 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:02.281 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:02.281 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:02.280+0000 7fa344ead0c0 0 bucket checkpoint complete 2020-10-09T17:20:02.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:02.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:02.313 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:20:02.361 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:20:02.361 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:20:02.555 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:02.556+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:02.556 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:02.556 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:07.560 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:07.560+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:07.560 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:07.561 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:12.564 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:12.564+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:12.564 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:12.564 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:17.568 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:17.568+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:17.568 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:17.568 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:22.572+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:22.572 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:22.572 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:27.576 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:27.576+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:27.576 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:27.576 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:32.580 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:32.580+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:32.581 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:32.581 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:37.584 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:37.584+0000 7f67ade9f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:37.585 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.19.10, , , , , ] 2020-10-09T17:20:37.585 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:42.589 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:42.588+0000 7f67ade9f0c0 1 bucket sync caught up with source: 2020-10-09T17:20:42.589 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000005.25.9, , , , , ] 2020-10-09T17:20:42.589 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.23.9, , , , , ] 2020-10-09T17:20:42.590 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:42.588+0000 7f67ade9f0c0 0 bucket checkpoint complete 2020-10-09T17:20:42.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:42.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:42.668 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:20:42.716 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:20:42.717 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:20:42.919 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:42.920+0000 7f26eba910c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:42.920 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , , , ] 2020-10-09T17:20:42.920 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:47.924 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:47.923+0000 7f26eba910c0 1 bucket sync caught up with source: 2020-10-09T17:20:47.924 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:47.924 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:47.925 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:20:47.923+0000 7f26eba910c0 0 bucket checkpoint complete 2020-10-09T17:20:47.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:47.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:20:47.956 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:20:48.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:20:48.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:20:48.236 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:20:48.238+0000 7f4db03270c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:48.236 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:48.236 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:20:53.241 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:20:53.241+0000 7f4db03270c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:53.241 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:53.241 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:20:58.245 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:20:58.244+0000 7f4db03270c0 1 waiting for incremental sync to catch up: 2020-10-09T17:20:58.246 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:20:58.246 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:21:03.250 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:03.247+0000 7f4db03270c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:03.250 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:21:03.251 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:21:08.255 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:08.254+0000 7f4db03270c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:08.255 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.15.10, , , , , ] 2020-10-09T17:21:08.256 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:21:13.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:13.258+0000 7f4db03270c0 1 bucket sync caught up with source: 2020-10-09T17:21:13.260 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:21:13.260 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000005.21.9, , , , , ] 2020-10-09T17:21:13.260 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:13.258+0000 7f4db03270c0 0 bucket checkpoint complete 2020-10-09T17:21:15.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.019 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:15.070 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:15.070 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:15.277 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:15.278+0000 7f7ed4a0d0c0 1 bucket sync caught up with source: 2020-10-09T17:21:15.278 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:15.278 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:15.278 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:15.278+0000 7f7ed4a0d0c0 0 bucket checkpoint complete 2020-10-09T17:21:15.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.307 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:15.355 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:15.356 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:15.587 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:15.582+0000 7fd094b450c0 1 bucket sync caught up with source: 2020-10-09T17:21:15.587 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:15.587 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:15.587 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:15.582+0000 7fd094b450c0 0 bucket checkpoint complete 2020-10-09T17:21:15.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:15.614 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:15.660 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:15.661 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:15.883 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:15.881+0000 7ffa40aec0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:15.884 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000008.19.11, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:15.884 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:20.888 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:20.885+0000 7ffa40aec0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:20.889 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000008.19.11, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:20.889 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:25.893 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:25.888+0000 7ffa40aec0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:25.894 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000008.19.11, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:25.894 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:30.904 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:30.900+0000 7ffa40aec0c0 1 bucket sync caught up with source: 2020-10-09T17:21:30.905 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:30.905 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:30.905 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:30.900+0000 7ffa40aec0c0 0 bucket checkpoint complete 2020-10-09T17:21:30.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:30.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:30.947 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:30.996 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:30.997 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:31.226 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:31.222+0000 7efe82cef0c0 1 bucket sync caught up with source: 2020-10-09T17:21:31.226 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.10.9, , 00000000010.31.5, , 00000000007.23.9, , , , , ] 2020-10-09T17:21:31.227 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.10.9, , 00000000010.31.5, , 00000000007.23.9, , , , , ] 2020-10-09T17:21:31.227 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:31.222+0000 7efe82cef0c0 0 bucket checkpoint complete 2020-10-09T17:21:31.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:31.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:31.256 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:31.303 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:31.304 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:31.531 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:31.526+0000 7f4a7ec500c0 1 bucket sync caught up with source: 2020-10-09T17:21:31.531 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.10.9, , 00000000010.31.5, , 00000000007.23.9, , , , , ] 2020-10-09T17:21:31.531 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.10.9, , 00000000010.31.5, , 00000000007.23.9, , , , , ] 2020-10-09T17:21:31.531 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:31.526+0000 7f4a7ec500c0 0 bucket checkpoint complete 2020-10-09T17:21:31.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:31.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:31.561 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:31.606 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:31.607 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:31.828 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:31.822+0000 7f2c93ddd0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:31.828 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.15.9, , 00000000002.14.10, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:31.829 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.15.9, , 00000000003.16.9, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:36.832 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:36.825+0000 7f2c93ddd0c0 1 bucket sync caught up with source: 2020-10-09T17:21:36.833 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.15.9, , 00000000003.16.9, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:36.833 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.15.9, , 00000000003.16.9, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:36.833 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:36.825+0000 7f2c93ddd0c0 0 bucket checkpoint complete 2020-10-09T17:21:36.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:36.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:36.864 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:36.911 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:36.912 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:37.143 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:37.140+0000 7fab4d65b0c0 1 bucket sync caught up with source: 2020-10-09T17:21:37.143 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.15.9, , 00000000003.16.9, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:37.143 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.15.9, , 00000000003.16.9, , 00000000014.35.5, , , , , ] 2020-10-09T17:21:37.144 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:37.140+0000 7fab4d65b0c0 0 bucket checkpoint complete 2020-10-09T17:21:37.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone2, test-zone1} 2020-10-09T17:21:37.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:37.191 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:37.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:37.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:37.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:37.193 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:37.199 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:37.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:37.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:37.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:37.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:37.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:37.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:37.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:37.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:37.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:37.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:37.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:37.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:37.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:38.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:38.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:38.222 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone2, test-zone1} 2020-10-09T17:21:38.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone3, test-zone1} 2020-10-09T17:21:38.232 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:38.237 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:38.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:38.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:38.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:38.238 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:38.244 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:38.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:38.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:38.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:38.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:38.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:38.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:38.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:38.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:38.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:38.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:38.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:38.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:38.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:39.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:39.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:39.237 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone3, test-zone1} 2020-10-09T17:21:39.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone3, test-zone2} 2020-10-09T17:21:39.248 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:39.254 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:39.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:39.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:39.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:39.255 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:39.262 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:39.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:39.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:39.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:39.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:39.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:39.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:39.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:39.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:39.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:39.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:39.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:39.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:39.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:40.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:40.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:40.234 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone3, test-zone2} 2020-10-09T17:21:40.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.236 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:40.283 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:40.283 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:40.514 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:40.507+0000 7f33a286d0c0 1 bucket sync caught up with source: 2020-10-09T17:21:40.514 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:40.514 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:40.514 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:40.507+0000 7f33a286d0c0 0 bucket checkpoint complete 2020-10-09T17:21:40.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.546 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:40.594 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:40.801+0000 7f1a2f8e50c0 1 bucket sync caught up with source: 2020-10-09T17:21:40.809 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:40.809 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:40.809 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:40.801+0000 7f1a2f8e50c0 0 bucket checkpoint complete 2020-10-09T17:21:40.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:40.837 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:40.886 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:40.886 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:41.125 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:41.119+0000 7fec01c360c0 1 bucket sync caught up with source: 2020-10-09T17:21:41.125 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:41.126 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.32.5, , 00000000003.13.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:41.126 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:41.119+0000 7fec01c360c0 0 bucket checkpoint complete 2020-10-09T17:21:41.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.256 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:41.305 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:41.535 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:41.529+0000 7f970b1110c0 1 bucket sync caught up with source: 2020-10-09T17:21:41.536 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.23.9, , 00000000010.34.5, , 00000000009.33.9, , , , , ] 2020-10-09T17:21:41.536 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.23.9, , 00000000010.34.5, , 00000000009.33.9, , , , , ] 2020-10-09T17:21:41.536 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:41.529+0000 7f970b1110c0 0 bucket checkpoint complete 2020-10-09T17:21:41.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.568 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:41.614 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:41.615 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:41.832 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:41.825+0000 7f838a7460c0 1 bucket sync caught up with source: 2020-10-09T17:21:41.832 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.23.9, , 00000000010.34.5, , 00000000009.33.9, , , , , ] 2020-10-09T17:21:41.832 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.23.9, , 00000000010.34.5, , 00000000009.33.9, , , , , ] 2020-10-09T17:21:41.833 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:41.825+0000 7f838a7460c0 0 bucket checkpoint complete 2020-10-09T17:21:41.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:41.863 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:41.907 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:42.140 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:42.133+0000 7fe48d4690c0 1 bucket sync caught up with source: 2020-10-09T17:21:42.140 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.21.9, , 00000000003.18.9, , 00000000014.45.5, , , , , ] 2020-10-09T17:21:42.141 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.21.9, , 00000000003.18.9, , 00000000014.45.5, , , , , ] 2020-10-09T17:21:42.141 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:42.133+0000 7fe48d4690c0 0 bucket checkpoint complete 2020-10-09T17:21:42.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:42.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:42.170 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:42.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:42.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:42.456 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:42.451+0000 7f2f729a80c0 1 bucket sync caught up with source: 2020-10-09T17:21:42.456 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.21.9, , 00000000003.18.9, , 00000000014.45.5, , , , , ] 2020-10-09T17:21:42.457 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.21.9, , 00000000003.18.9, , 00000000014.45.5, , , , , ] 2020-10-09T17:21:42.457 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:42.451+0000 7f2f729a80c0 0 bucket checkpoint complete 2020-10-09T17:21:42.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone2, test-zone1} 2020-10-09T17:21:42.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:42.504 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:42.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:42.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:42.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:42.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:42.512 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:42.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:42.513 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:42.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:42.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:42.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:42.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:42.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:42.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:42.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:42.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:43.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:43.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:43.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:43.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:43.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:43.532 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone2, test-zone1} 2020-10-09T17:21:43.532 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone3, test-zone1} 2020-10-09T17:21:43.540 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:43.545 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:43.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:43.546 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:43.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:43.547 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:43.552 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:43.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:43.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:43.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:43.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:43.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:43.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:43.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:43.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:43.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:43.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:44.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:44.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:44.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:44.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:44.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:44.529 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone3, test-zone1} 2020-10-09T17:21:44.530 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone3, test-zone2} 2020-10-09T17:21:44.541 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:44.547 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:44.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:44.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:44.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:44.549 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:44.558 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:44.558 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:44.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:44.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:44.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:44.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:44.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:44.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:44.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:44.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:44.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:45.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:45.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:45.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:45.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:45.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:45.554 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone3, test-zone2} 2020-10-09T17:21:45.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:45.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:45.556 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:45.600 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:45.601 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:45.830 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:45.823+0000 7f96576e80c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:45.830 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.28.5, , 00000000003.14.9, , 00000000004.14.9, , , , , ] 2020-10-09T17:21:45.831 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:50.834 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:50.827+0000 7f96576e80c0 1 bucket sync caught up with source: 2020-10-09T17:21:50.834 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:50.834 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:50.834 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:50.827+0000 7f96576e80c0 0 bucket checkpoint complete 2020-10-09T17:21:50.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:50.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:50.867 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:50.914 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:50.915 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:51.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:51.129+0000 7fdc0a0200c0 1 bucket sync caught up with source: 2020-10-09T17:21:51.134 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:51.135 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:51.135 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:51.129+0000 7fdc0a0200c0 0 bucket checkpoint complete 2020-10-09T17:21:51.160 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.161 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:51.208 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:51.208 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:51.462 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:51.455+0000 7f5acf6d30c0 1 bucket sync caught up with source: 2020-10-09T17:21:51.463 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:51.463 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000010.28.5, , 00000000003.14.9, , 00000000007.20.9, , , , , ] 2020-10-09T17:21:51.463 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:51.455+0000 7f5acf6d30c0 0 bucket checkpoint complete 2020-10-09T17:21:51.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.584 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:51.633 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:51.634 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:51.856 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:51.849+0000 7fb1692090c0 1 bucket sync caught up with source: 2020-10-09T17:21:51.856 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.18.9, , 00000000010.35.5, , 00000000007.22.9, , , , , ] 2020-10-09T17:21:51.856 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.18.9, , 00000000010.35.5, , 00000000007.22.9, , , , , ] 2020-10-09T17:21:51.857 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:51.849+0000 7fb1692090c0 0 bucket checkpoint complete 2020-10-09T17:21:51.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:51.888 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:51.934 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:51.935 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:52.164 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:52.157+0000 7f336cdde0c0 1 bucket sync caught up with source: 2020-10-09T17:21:52.164 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.18.9, , 00000000010.35.5, , 00000000007.22.9, , , , , ] 2020-10-09T17:21:52.164 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.18.9, , 00000000010.35.5, , 00000000007.22.9, , , , , ] 2020-10-09T17:21:52.164 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:52.157+0000 7f336cdde0c0 0 bucket checkpoint complete 2020-10-09T17:21:52.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:52.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:52.195 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:52.243 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:52.244 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:52.469+0000 7ff01123e0c0 1 bucket sync caught up with source: 2020-10-09T17:21:52.476 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000003.17.9, , 00000000003.19.9, , 00000000016.43.5, , , , , ] 2020-10-09T17:21:52.476 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000003.17.9, , 00000000003.19.9, , 00000000016.43.5, , , , , ] 2020-10-09T17:21:52.476 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:21:52.469+0000 7ff01123e0c0 0 bucket checkpoint complete 2020-10-09T17:21:52.504 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:52.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:52.505 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:52.553 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:52.553 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:52.779 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:52.775+0000 7f1be11600c0 1 bucket sync caught up with source: 2020-10-09T17:21:52.779 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000003.17.9, , 00000000003.19.9, , 00000000016.43.5, , , , , ] 2020-10-09T17:21:52.779 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000003.17.9, , 00000000003.19.9, , 00000000016.43.5, , , , , ] 2020-10-09T17:21:52.780 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:52.775+0000 7f1be11600c0 0 bucket checkpoint complete 2020-10-09T17:21:52.810 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone2, test-zone1} 2020-10-09T17:21:52.821 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:52.827 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:52.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:52.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:52.829 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:52.837 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:52.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:52.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:52.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:52.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:52.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:52.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:53.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:53.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:53.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:53.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:53.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:53.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:53.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:53.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:53.858 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone2, test-zone1} 2020-10-09T17:21:53.859 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone3, test-zone1} 2020-10-09T17:21:53.867 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:53.873 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:53.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:53.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:53.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:53.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:53.880 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:53.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:53.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:53.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:53.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:53.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:53.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:54.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:54.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:54.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:54.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:54.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:54.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:54.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:54.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:54.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:54.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone3, test-zone1} 2020-10-09T17:21:54.854 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone3, test-zone2} 2020-10-09T17:21:54.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:21:54.872 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:54.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:54.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:54.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:54.873 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:21:54.879 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:21:54.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:21:54.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:21:54.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:21:54.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:54.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:54.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:21:55.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:55.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:55.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:21:55.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:55.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:55.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:21:55.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:55.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:55.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:21:55.867 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone3, test-zone2} 2020-10-09T17:21:56.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:56.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:21:56.481 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:21:56.527 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:21:56.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:21:56.768 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:21:56.766+0000 7f878370f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:21:56.769 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:21:56.769 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:01.772 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:01.770+0000 7f878370f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:01.773 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:01.773 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:06.776 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:06.774+0000 7f878370f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:06.777 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000010.23.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:06.777 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:11.781 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:11.778+0000 7f878370f0c0 1 bucket sync caught up with source: 2020-10-09T17:22:11.781 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.30.5, , 00000000003.11.9, , 00000000010.25.5, , , , , ] 2020-10-09T17:22:11.782 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000009.21.9, , , , , ] 2020-10-09T17:22:11.782 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:11.778+0000 7f878370f0c0 0 bucket checkpoint complete 2020-10-09T17:22:11.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:11.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:11.813 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:11.861 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:12.090 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:12.084+0000 7fbd13cff0c0 1 bucket sync caught up with source: 2020-10-09T17:22:12.090 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000011.30.5, , 00000000003.11.9, , 00000000010.25.5, , , , , ] 2020-10-09T17:22:12.091 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000010.25.5, , , , , ] 2020-10-09T17:22:12.091 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:12.084+0000 7fbd13cff0c0 0 bucket checkpoint complete 2020-10-09T17:22:12.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:12.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:12.117 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:12.164 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:12.164 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:12.407 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:12.402+0000 7fbd66b650c0 1 bucket sync caught up with source: 2020-10-09T17:22:12.407 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.30.5, , 00000000003.11.9, , 00000000010.25.5, , , , , ] 2020-10-09T17:22:12.407 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.30.5, , 00000000003.11.9, , 00000000010.25.5, , , , , ] 2020-10-09T17:22:12.407 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:12.402+0000 7fbd66b650c0 0 bucket checkpoint complete 2020-10-09T17:22:12.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:12.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:12.534 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:12.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:12.793 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:12.788+0000 7f3b49bad0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:12.793 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.13.5, , 00000000010.31.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:12.793 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:17.797 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:17.792+0000 7f3b49bad0c0 1 bucket sync caught up with source: 2020-10-09T17:22:17.798 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:17.798 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:17.798 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:17.792+0000 7f3b49bad0c0 0 bucket checkpoint complete 2020-10-09T17:22:17.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:17.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:17.829 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:17.874 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:18.095 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:18.092+0000 7fb63e4790c0 1 bucket sync caught up with source: 2020-10-09T17:22:18.095 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:18.096 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.13.5, , 00000000011.37.5, , 00000000008.26.5, , , , , ] 2020-10-09T17:22:18.096 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:18.092+0000 7fb63e4790c0 0 bucket checkpoint complete 2020-10-09T17:22:18.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:18.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:18.128 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:18.215 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:18.216 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:18.446 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:18.444+0000 7f5afcc740c0 1 bucket sync caught up with source: 2020-10-09T17:22:18.447 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.18.5, , 00000000004.19.5, , 00000000015.38.5, , , , , ] 2020-10-09T17:22:18.447 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.18.5, , 00000000004.19.5, , 00000000015.38.5, , , , , ] 2020-10-09T17:22:18.447 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:18.444+0000 7f5afcc740c0 0 bucket checkpoint complete 2020-10-09T17:22:18.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:18.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:18.475 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:18.522 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:18.523 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:18.757 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:18.754+0000 7f89873360c0 1 bucket sync caught up with source: 2020-10-09T17:22:18.825 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.18.5, , 00000000004.19.5, , 00000000015.38.5, , , , , ] 2020-10-09T17:22:18.826 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.18.5, , 00000000004.19.5, , 00000000015.38.5, , , , , ] 2020-10-09T17:22:18.826 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:18.754+0000 7f89873360c0 0 bucket checkpoint complete 2020-10-09T17:22:18.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone2, test-zone1} 2020-10-09T17:22:18.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:18.845 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:18.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:18.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:18.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:18.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:18.853 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:18.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:18.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:18.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:18.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:18.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:18.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:19.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:19.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:19.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:19.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:19.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:19.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:19.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:19.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:19.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:19.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone2, test-zone1} 2020-10-09T17:22:19.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone3, test-zone1} 2020-10-09T17:22:19.994 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:20.005 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:20.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:20.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:20.006 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:20.006 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:20.019 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:20.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:20.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:20.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:20.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:20.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:20.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:20.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:20.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:20.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:20.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:20.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:20.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:20.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:21.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:21.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:21.248 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone3, test-zone1} 2020-10-09T17:22:21.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-16 zones={test-zone3, test-zone2} 2020-10-09T17:22:21.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:21.265 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:21.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:21.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:21.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:21.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:21.273 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:21.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:21.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:21.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:21.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:21.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:21.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:21.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:21.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:21.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:21.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:21.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:21.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:22.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:22.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:22.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:22.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-16 zones={test-zone3, test-zone2} 2020-10-09T17:22:22.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.288 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:22.333 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:22.333 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:22.568 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:22.565+0000 7f0d0f7700c0 1 bucket sync caught up with source: 2020-10-09T17:22:22.568 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:22.569 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:22.569 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:22.565+0000 7f0d0f7700c0 0 bucket checkpoint complete 2020-10-09T17:22:22.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.600 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:22.650 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:22.870 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:22.864+0000 7f7e5b3db0c0 1 bucket sync caught up with source: 2020-10-09T17:22:22.870 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:22.870 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:22.870 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:22.864+0000 7f7e5b3db0c0 0 bucket checkpoint complete 2020-10-09T17:22:22.897 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:22.898 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:22.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:22.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:23.189 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:23.185+0000 7f6ca8c8b0c0 1 bucket sync caught up with source: 2020-10-09T17:22:23.189 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:23.189 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.39.5, , 00000000004.16.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:23.190 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:23.185+0000 7f6ca8c8b0c0 0 bucket checkpoint complete 2020-10-09T17:22:23.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.221 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:23.497 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:23.492+0000 7f34c73030c0 1 bucket sync caught up with source: 2020-10-09T17:22:23.498 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.26.5, , 00000000011.41.5, , 00000000010.36.5, , , , , ] 2020-10-09T17:22:23.498 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.26.5, , 00000000011.41.5, , 00000000010.36.5, , , , , ] 2020-10-09T17:22:23.498 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:23.492+0000 7f34c73030c0 0 bucket checkpoint complete 2020-10-09T17:22:23.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.529 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:23.575 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:23.575 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:23.787 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:23.784+0000 7faf5aad60c0 1 bucket sync caught up with source: 2020-10-09T17:22:23.788 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.26.5, , 00000000011.41.5, , 00000000010.36.5, , , , , ] 2020-10-09T17:22:23.788 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.26.5, , 00000000011.41.5, , 00000000010.36.5, , , , , ] 2020-10-09T17:22:23.788 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:23.784+0000 7faf5aad60c0 0 bucket checkpoint complete 2020-10-09T17:22:23.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:23.819 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:23.868 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:24.113 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:24.108+0000 7feafa6810c0 1 bucket sync caught up with source: 2020-10-09T17:22:24.114 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.24.5, , 00000000004.21.5, , 00000000015.48.5, , , , , ] 2020-10-09T17:22:24.114 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.24.5, , 00000000004.21.5, , 00000000015.48.5, , , , , ] 2020-10-09T17:22:24.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:24.108+0000 7feafa6810c0 0 bucket checkpoint complete 2020-10-09T17:22:24.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:24.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:24.141 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:24.193 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:24.194 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:24.413 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:24.409+0000 7f62285ea0c0 1 bucket sync caught up with source: 2020-10-09T17:22:24.413 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.24.5, , 00000000004.21.5, , 00000000015.48.5, , , , , ] 2020-10-09T17:22:24.413 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.24.5, , 00000000004.21.5, , 00000000015.48.5, , , , , ] 2020-10-09T17:22:24.413 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:24.409+0000 7f62285ea0c0 0 bucket checkpoint complete 2020-10-09T17:22:24.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone2, test-zone1} 2020-10-09T17:22:24.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:24.462 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:24.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:24.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:24.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:24.465 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:24.472 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:24.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:24.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:24.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:24.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:24.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:24.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:24.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:24.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:24.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:24.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:24.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:25.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:25.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:25.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:25.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:25.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone2, test-zone1} 2020-10-09T17:22:25.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone3, test-zone1} 2020-10-09T17:22:25.496 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:25.501 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:25.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:25.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:25.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:25.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:25.514 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:25.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:25.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:25.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:25.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:25.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:25.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:25.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:25.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:25.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:25.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:26.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:26.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:26.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:26.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:26.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:26.501 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone3, test-zone1} 2020-10-09T17:22:26.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-17 zones={test-zone3, test-zone2} 2020-10-09T17:22:26.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:26.517 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:26.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:26.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:26.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:26.519 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:26.525 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:26.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:26.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:26.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:26.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:26.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:26.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:26.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:26.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:26.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:26.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:27.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:27.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:27.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:27.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:27.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:27.525 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-17 zones={test-zone3, test-zone2} 2020-10-09T17:22:27.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:27.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:27.528 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:27.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:27.575 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:27.814 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:27.809+0000 7f7a4a5af0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:27.814 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.35.5, , 00000000004.17.5, , 00000000007.20.9, , , , , ] 2020-10-09T17:22:27.814 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:32.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:32.813+0000 7f7a4a5af0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:32.818 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.35.5, , 00000000004.17.5, , 00000000007.20.9, , , , , ] 2020-10-09T17:22:32.818 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:37.823 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:37.817+0000 7f7a4a5af0c0 1 bucket sync caught up with source: 2020-10-09T17:22:37.823 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:37.823 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:37.824 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:37.817+0000 7f7a4a5af0c0 0 bucket checkpoint complete 2020-10-09T17:22:37.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:37.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:37.851 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:37.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:37.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:38.128 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:38.123+0000 7f023a1010c0 1 bucket sync caught up with source: 2020-10-09T17:22:38.128 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:38.128 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:38.128 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:38.123+0000 7f023a1010c0 0 bucket checkpoint complete 2020-10-09T17:22:38.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.157 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:38.205 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:38.206 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:38.449 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:38.445+0000 7f04074860c0 1 bucket sync caught up with source: 2020-10-09T17:22:38.450 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:38.450 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000011.35.5, , 00000000004.17.5, , 00000000008.23.5, , , , , ] 2020-10-09T17:22:38.450 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:38.445+0000 7f04074860c0 0 bucket checkpoint complete 2020-10-09T17:22:38.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.575 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:38.624 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:38.625 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:38.843 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:38.839+0000 7f11a23580c0 1 bucket sync caught up with source: 2020-10-09T17:22:38.843 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.21.5, , 00000000011.41.5, , 00000000008.25.5, , , , , ] 2020-10-09T17:22:38.844 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.21.5, , 00000000011.41.5, , 00000000008.25.5, , , , , ] 2020-10-09T17:22:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:38.839+0000 7f11a23580c0 0 bucket checkpoint complete 2020-10-09T17:22:38.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:38.867 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:38.913 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:38.914 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:39.135+0000 7f9bae4cd0c0 1 bucket sync caught up with source: 2020-10-09T17:22:39.140 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.21.5, , 00000000011.41.5, , 00000000008.25.5, , , , , ] 2020-10-09T17:22:39.141 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.21.5, , 00000000011.41.5, , 00000000008.25.5, , , , , ] 2020-10-09T17:22:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:39.135+0000 7f9bae4cd0c0 0 bucket checkpoint complete 2020-10-09T17:22:39.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:39.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:39.174 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:39.263 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:39.263 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:39.492 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:39.487+0000 7fb8216a20c0 1 bucket sync caught up with source: 2020-10-09T17:22:39.493 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, 00000000004.20.5, , 00000000004.22.5, , 00000000017.46.5, , , , , ] 2020-10-09T17:22:39.493 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, 00000000004.20.5, , 00000000004.22.5, , 00000000017.46.5, , , , , ] 2020-10-09T17:22:39.493 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:39.487+0000 7fb8216a20c0 0 bucket checkpoint complete 2020-10-09T17:22:39.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:39.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:39.525 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:39.572 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:39.573 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:39.812 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:39.809+0000 7fbae6a7e0c0 1 bucket sync caught up with source: 2020-10-09T17:22:39.812 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, 00000000004.20.5, , 00000000004.22.5, , 00000000017.46.5, , , , , ] 2020-10-09T17:22:39.812 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, 00000000004.20.5, , 00000000004.22.5, , 00000000017.46.5, , , , , ] 2020-10-09T17:22:39.813 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:39.809+0000 7fbae6a7e0c0 0 bucket checkpoint complete 2020-10-09T17:22:39.840 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone2, test-zone1} 2020-10-09T17:22:39.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:39.857 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:39.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:39.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:39.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:39.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:39.867 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:39.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:39.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:39.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:39.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:39.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:39.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:40.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:40.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:40.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:40.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:40.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:40.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:40.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:40.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:40.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:40.874 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone2, test-zone1} 2020-10-09T17:22:40.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone3, test-zone1} 2020-10-09T17:22:40.884 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:40.889 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:40.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:40.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:40.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:40.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:40.896 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:40.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:40.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:40.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:40.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:40.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:40.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:41.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:41.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:41.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:41.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:41.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:41.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:41.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:41.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:41.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:41.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone3, test-zone1} 2020-10-09T17:22:41.906 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-18 zones={test-zone3, test-zone2} 2020-10-09T17:22:41.923 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:41.929 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:41.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:41.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:41.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:41.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:41.938 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:41.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-10-09T17:22:41.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-10-09T17:22:41.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-10-09T17:22:41.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:41.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:42.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:42.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:42.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:42.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-10-09T17:22:42.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:42.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:42.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-10-09T17:22:42.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:42.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:42.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-10-09T17:22:42.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-18 zones={test-zone3, test-zone2} 2020-10-09T17:22:42.959 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-10-09T17:22:42.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.001 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:22:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:43.236 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:22:43.270 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:22:43.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:22:43.271 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.272 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264162.996213_169.1", 2020-10-09T17:22:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:42.996213Z" 2020-10-09T17:22:43.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.273 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:22:43.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:22:43.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.277 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:22:43.323 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:43.324 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:43.572 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:22:43.572 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:22:43.572 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:22:43.572 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:22:43.572 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:22:43.573 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:43.574 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:43.574 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:22:43.574 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:43.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:22:43.574 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:22:43.575 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:22:43.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:43.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:43.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264162.996213_169.1", 2020-10-09T17:22:43.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:43.580 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:43.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:43.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.996213Z", 2020-10-09T17:22:43.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:22:43.581 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:22:43.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:43.582 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:22:43.583 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:22:43.584 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:22:43.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.585 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:22:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:43.625 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:43.847 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:22:43.848 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:22:43.849 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:22:43.851 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:22:43.852 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:22:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:22:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:22:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:22:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:22:43.853 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:22:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:22:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:22:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264162.996213_169.1", 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:22:43.855 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.996213Z", 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:22:43.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:22:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:22:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:22:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:22:43.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:22:43.857 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:22:43.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:22:43.859 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:22:43.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:22:43.860 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:22:43.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:43.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:44.170 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:22:44.170 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:22:44.170 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:22:44.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:22:44.171 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:22:44.171 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:22:44.171 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:44.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:44.172 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:22:44.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:44.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:22:44.173 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:44.173 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:44.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:22:44.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:44.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:22:44.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:44.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:22:44.175 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:44.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:44.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:44.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:22:44.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:44.176 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:44.176 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:22:44.177 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:44.177 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:44.177 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:44.177 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:22:44.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:44.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:44.178 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:44.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:22:44.179 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:44.179 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:44.180 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:22:44.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:44.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:44.180 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:44.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:22:44.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:44.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:44.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:22:44.182 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:22:44.182 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:22:44.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:22:44.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264162.996213_169.1", 2020-10-09T17:22:44.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:22:44.183 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:22:44.184 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:22:44.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.996213Z", 2020-10-09T17:22:44.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:22:44.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:22:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:22:44.185 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:22:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:22:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:22:44.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:22:44.187 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:22:44.188 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:22:44.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:44.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:44.248 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:44.298 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:44.298 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:44.528 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:44.525+0000 7f403eb910c0 1 bucket sync caught up with source: 2020-10-09T17:22:44.529 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2020-10-09T17:22:44.529 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000002.11.10, , , , , ] 2020-10-09T17:22:44.529 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:44.525+0000 7f403eb910c0 0 bucket checkpoint complete 2020-10-09T17:22:44.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:44.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:44.556 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:44.610 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:44.611 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:44.803 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:44.795+0000 7fc1cdf740c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:44.803 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2020-10-09T17:22:44.803 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2020-10-09T17:22:49.806 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:49.799+0000 7fc1cdf740c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:49.806 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000002.11.10, , , , , ] 2020-10-09T17:22:49.807 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2020-10-09T17:22:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:54.803+0000 7fc1cdf740c0 1 bucket sync caught up with source: 2020-10-09T17:22:54.809 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000005.17.9, , , , , ] 2020-10-09T17:22:54.809 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000005.17.9, , , , , ] 2020-10-09T17:22:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:54.803+0000 7fc1cdf740c0 0 bucket checkpoint complete 2020-10-09T17:22:54.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:54.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:54.833 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:54.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:54.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:55.106 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:55.096+0000 7f364cab20c0 1 bucket sync caught up with source: 2020-10-09T17:22:55.107 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , , , ] 2020-10-09T17:22:55.107 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , , , ] 2020-10-09T17:22:55.107 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:55.096+0000 7f364cab20c0 0 bucket checkpoint complete 2020-10-09T17:22:55.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.155 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:55.204 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:55.439+0000 7f6a610020c0 1 bucket sync caught up with source: 2020-10-09T17:22:55.445 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-10-09T17:22:55.445 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-10-09T17:22:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:55.439+0000 7f6a610020c0 0 bucket checkpoint complete 2020-10-09T17:22:55.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.479 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:55.528 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:55.529 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:55.748 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:55.743+0000 7f2cff2ce0c0 1 bucket sync caught up with source: 2020-10-09T17:22:55.749 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.18.9, , , , , ] 2020-10-09T17:22:55.749 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.18.9, , , , , ] 2020-10-09T17:22:55.749 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:55.743+0000 7f2cff2ce0c0 0 bucket checkpoint complete 2020-10-09T17:22:55.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:55.779 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:55.825 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:55.825 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:56.049 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:56.043+0000 7fdce69eb0c0 1 bucket sync caught up with source: 2020-10-09T17:22:56.049 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-10-09T17:22:56.049 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-10-09T17:22:56.049 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:22:56.043+0000 7fdce69eb0c0 0 bucket checkpoint complete 2020-10-09T17:22:56.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:56.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:56.078 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:56.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:56.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:56.370 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:56.364+0000 7f637b6580c0 1 bucket sync caught up with source: 2020-10-09T17:22:56.371 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.19.9, , , , , ] 2020-10-09T17:22:56.371 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.19.9, , , , , ] 2020-10-09T17:22:56.371 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:56.364+0000 7f637b6580c0 0 bucket checkpoint complete 2020-10-09T17:22:56.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone2, test-zone1} 2020-10-09T17:22:56.413 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:56.466 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:56.467 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:56.475 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:56.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone2, test-zone1} 2020-10-09T17:22:56.675 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone3, test-zone1} 2020-10-09T17:22:56.684 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:56.730 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:56.731 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:56.737 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:56.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:56.933 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone3, test-zone1} 2020-10-09T17:22:56.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone3, test-zone2} 2020-10-09T17:22:56.946 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:22:56.994 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:56.995 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:22:57.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:22:57.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:57.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:57.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:22:57.201 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone3, test-zone2} 2020-10-09T17:22:58.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:58.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:22:58.712 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:22:58.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:22:58.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:22:58.985 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:22:58.980+0000 7f37198780c0 1 waiting for incremental sync to catch up: 2020-10-09T17:22:58.985 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000004.12.11, , ] 2020-10-09T17:22:58.986 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:03.989 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:03.984+0000 7f37198780c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:03.990 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000004.12.11, , ] 2020-10-09T17:23:03.990 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:08.993 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:08.988+0000 7f37198780c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:08.994 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000004.12.11, , ] 2020-10-09T17:23:08.994 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:13.997 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:13.992+0000 7f37198780c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:13.998 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000004.12.11, , ] 2020-10-09T17:23:13.998 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:19.001 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:18.996+0000 7f37198780c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:19.002 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000005.17.9, , , 00000000039.106.5, , ] 2020-10-09T17:23:19.002 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:24.006 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:24.000+0000 7f37198780c0 1 bucket sync caught up with source: 2020-10-09T17:23:24.007 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000039.106.5, , ] 2020-10-09T17:23:24.007 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000026.66.11, , ] 2020-10-09T17:23:24.007 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:24.000+0000 7f37198780c0 0 bucket checkpoint complete 2020-10-09T17:23:24.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:23:24.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:23:24.039 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:23:24.087 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:23:24.088 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:23:24.323 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:23:24.319+0000 7f55a34bd0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:24.323 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000042.115.5, , ] 2020-10-09T17:23:24.323 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:29.327 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:23:29.323+0000 7f55a34bd0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:29.328 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000042.115.5, , ] 2020-10-09T17:23:29.328 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:34.331 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:23:34.327+0000 7f55a34bd0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:34.331 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000042.115.5, , ] 2020-10-09T17:23:34.331 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:39.335 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:23:39.331+0000 7f55a34bd0c0 1 bucket sync caught up with source: 2020-10-09T17:23:39.336 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:39.336 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:39.336 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:23:39.331+0000 7f55a34bd0c0 0 bucket checkpoint complete 2020-10-09T17:23:39.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:23:39.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:23:39.368 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:23:39.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:23:39.415 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:23:39.659 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:39.652+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:39.659 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:39.659 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:23:44.663 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:44.655+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:44.663 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:44.663 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:23:49.667 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:49.659+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:49.668 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:49.668 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:23:54.671 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:54.663+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:54.671 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:54.671 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:23:59.674 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:23:59.667+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:23:59.675 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:23:59.675 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:04.678 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:04.671+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:24:04.678 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:24:04.679 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:09.681 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:09.675+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:24:09.682 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:24:09.682 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:14.686 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:14.679+0000 7ffac815e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:24:14.686 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000045.121.9, , ] 2020-10-09T17:24:14.687 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:19.690 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:19.683+0000 7ffac815e0c0 1 bucket sync caught up with source: 2020-10-09T17:24:19.690 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:19.690 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000006.19.9, , , 00000000046.123.9, , ] 2020-10-09T17:24:19.690 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:19.683+0000 7ffac815e0c0 0 bucket checkpoint complete 2020-10-09T17:24:19.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:19.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:19.828 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:19.869 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:19.870 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:20.095 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.091+0000 7f57122e00c0 1 bucket sync caught up with source: 2020-10-09T17:24:20.096 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000050.128.9, , ] 2020-10-09T17:24:20.096 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000050.128.9, , ] 2020-10-09T17:24:20.096 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.091+0000 7f57122e00c0 0 bucket checkpoint complete 2020-10-09T17:24:20.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.127 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:20.219 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:20.219 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:20.452 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.447+0000 7f7b5d3b60c0 1 bucket sync caught up with source: 2020-10-09T17:24:20.452 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.18.9, , , 00000000050.128.9, , ] 2020-10-09T17:24:20.453 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.18.9, , , 00000000050.128.9, , ] 2020-10-09T17:24:20.453 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.447+0000 7f7b5d3b60c0 0 bucket checkpoint complete 2020-10-09T17:24:20.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.483 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:20.528 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:20.529 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:20.744 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.739+0000 7fcd764730c0 1 bucket sync caught up with source: 2020-10-09T17:24:20.744 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000051.126.9, , ] 2020-10-09T17:24:20.745 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000051.126.9, , ] 2020-10-09T17:24:20.745 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:20.739+0000 7fcd764730c0 0 bucket checkpoint complete 2020-10-09T17:24:20.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:20.773 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:20.819 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:20.820 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:21.042 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:21.035+0000 7efc54c0d0c0 1 bucket sync caught up with source: 2020-10-09T17:24:21.042 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000004.19.9, , , 00000000051.126.9, , ] 2020-10-09T17:24:21.043 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000004.19.9, , , 00000000051.126.9, , ] 2020-10-09T17:24:21.043 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:21.035+0000 7efc54c0d0c0 0 bucket checkpoint complete 2020-10-09T17:24:21.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone2, test-zone1} 2020-10-09T17:24:21.089 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:24:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.100 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.101 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.102 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:24:21.115 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:21.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:21.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:21.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:21.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:21.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:21.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:22.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:23.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:24.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:25.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:26.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:27.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:28.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:29.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone2, test-zone1} 2020-10-09T17:24:30.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone3, test-zone1} 2020-10-09T17:24:30.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:24:30.212 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:30.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.219 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:24:30.229 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:30.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.231 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:30.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:30.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:30.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:30.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:30.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:31.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:32.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:33.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:34.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:35.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:36.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:37.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:38.801 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone3, test-zone1} 2020-10-09T17:24:38.801 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-19 zones={test-zone3, test-zone2} 2020-10-09T17:24:38.812 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:24:38.821 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:38.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.823 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.824 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.825 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.826 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:24:38.840 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-10-09T17:24:38.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:24:38.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:38.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:38.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-10-09T17:24:39.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:39.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:40.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:41.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:42.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:43.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:44.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:45.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:46.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:24:47.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-19 zones={test-zone3, test-zone2} 2020-10-09T17:24:47.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:47.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:47.884 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:47.923 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:47.924 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:48.173 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.174 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:48.176 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:48.176 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:48.177 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:24:48.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.178 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:48.219 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:48.219 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:24:48.476 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:48.477 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-10-09T17:24:48.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:48.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:24:48.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:48.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:48.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:48.483 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:24:48.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:24:48.484 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:24:48.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:48.484 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:24:48.485 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:24:48.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.486 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:48.526 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:48.526 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:48.772 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:48.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.772 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:48.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:48.774 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:48.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:48.775 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:48.776 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:24:48.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:48.777 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:48.861 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:48.862 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:49.109 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:24:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:24:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:24:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:24:49.109 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 2964130218513816294 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:49.110 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000" 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:49.112 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:24:49.113 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:49.114 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:49.115 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.116 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:24:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:24:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:24:49.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.117 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:24:49.118 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:24:49.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.119 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:49.457 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:49.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.458 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:49.459 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:49.460 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T17:24:49.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.462 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:49.504 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:49.504 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:49.772 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:24:49.772 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:24:49.772 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:49.773 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:49.774 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.775 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:49.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:49.777 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:00.601753Z" 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:49.778 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.779 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:49.780 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:24:49.781 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T17:24:49.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:49.783 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:49.862 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:49.863 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:50.097 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:24:50.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:50.097 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:24:00.682093Z" 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:50.099 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:24:50.100 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-09T17:24:50.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.102 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:50.142 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:50.142 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:50.384 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:24:50.385 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:24:50.385 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:24:50.385 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:24:50.385 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:24:50.385 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 13602710843643633186 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.386 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000" 2020-10-09T17:24:50.387 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:24:50.388 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.390 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:00.682093Z" 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:24:50.391 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.393 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:24:50.394 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-09T17:24:50.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.396 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:50.437 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:50.438 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:50.696 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:24:00.682093Z" 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:50.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:50.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:24:50.699 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:24:50.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:50.699 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:24:50.700 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-09T17:24:50.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.701 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:50.746 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:50.747 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:50.983 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:24:50.984 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:24:50.984 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:24:50.984 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:24:50.984 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:24:50.984 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 743555352260775809 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.985 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000" 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.987 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.988 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:00.682093Z" 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:50.990 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:24:50.991 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:24:50.992 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:50.992 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:24:50.993 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-09T17:24:50.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:50.994 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:51.033 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:51.033 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:51.284 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:51.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.285 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:00.623189Z" 2020-10-09T17:24:51.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:24:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:24:51.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.287 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:51.288 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-09T17:24:51.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.289 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:51.372 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:51.373 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:51.627 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:24:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:24:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:24:51.627 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 6335292525355606852 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.628 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000" 2020-10-09T17:24:51.629 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:51.630 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:24:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:24:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:00.623189Z" 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:24:51.634 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:24:51.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:24:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:24:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:24:51.636 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.636 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:24:51.637 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-09T17:24:51.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.639 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:51.725 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:51.726 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:51.958 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:51.959 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:00.623189Z" 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:51.960 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:24:51.961 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:24:51.961 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:51.961 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:51.962 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-09T17:24:51.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:51.964 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:52.006 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:52.007 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:52.265 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:24:52.265 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:24:52.266 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:24:52.266 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:24:52.266 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:24:52.266 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 601039138999268403 2020-10-09T17:24:52.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:52.267 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000000", 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000" 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.269 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:24:52.269 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.269 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:52.269 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:24:52.270 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.270 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.270 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.270 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:24:52.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.271 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:24:52.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:52.272 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000001310", 2020-10-09T17:24:52.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:00.623189Z" 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.275 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:24:52.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:24:52.276 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:24:52.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.276 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:24:52.277 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-09T17:24:52.278 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-10-09T17:24:52.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.312 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:52.346 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:52.347 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:52.558 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:24:52.577 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:24:52.577 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:52.577 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:24:52.578 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:24:52.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:52.578 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:52.578 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:24:52.578 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:52.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:52.580 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264292.305660_193.1", 2020-10-09T17:24:52.580 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:24:52.305660Z" 2020-10-09T17:24:52.580 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:52.580 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:24:52.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:24:52.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.582 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:52.618 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:52.619 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:52.862 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:24:52.862 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:24:52.862 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:24:52.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:24:52.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:24:52.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:24:52.866 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:24:52.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264292.305660_193.1", 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:52.869 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.305660Z", 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:24:52.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:24:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:24:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:24:52.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:52.871 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:24:52.872 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:24:52.873 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:24:52.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:52.874 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:52.915 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:52.916 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:53.166 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:24:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:24:53.166 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:53.167 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:24:53.168 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:24:53.169 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:24:53.170 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:24:53.171 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:24:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:24:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:24:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:24:53.173 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264292.305660_193.1", 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.305660Z", 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.174 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:24:53.175 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:24:53.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:24:53.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:24:53.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:53.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:24:53.178 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:24:53.218 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:53.219 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:53.480 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:24:53.480 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:24:53.480 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:24:53.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:53.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:53.483 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:53.484 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:53.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264292.305660_193.1", 2020-10-09T17:24:53.487 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.305660Z", 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.488 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:24:53.489 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:24:53.490 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:24:53.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:53.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:53.543 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:53.591 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:53.592 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:53.825 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:53.822+0000 7fecd7dab0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:24:53.826 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , , , ] 2020-10-09T17:24:53.826 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:58.829 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:58.826+0000 7fecd7dab0c0 1 bucket sync caught up with source: 2020-10-09T17:24:58.830 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:58.830 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:58.830 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:58.826+0000 7fecd7dab0c0 0 bucket checkpoint complete 2020-10-09T17:24:58.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:58.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:58.862 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:58.912 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:59.140 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:59.134+0000 7fe6640350c0 1 bucket sync caught up with source: 2020-10-09T17:24:59.140 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:59.141 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:59.141 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:59.134+0000 7fe6640350c0 0 bucket checkpoint complete 2020-10-09T17:24:59.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.168 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:59.215 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:59.216 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:59.449 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:59.446+0000 7ffadfe900c0 1 bucket sync caught up with source: 2020-10-09T17:24:59.449 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:59.449 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:24:59.450 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:24:59.446+0000 7ffadfe900c0 0 bucket checkpoint complete 2020-10-09T17:24:59.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.582 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:59.630 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:59.631 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:24:59.853 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:59.846+0000 7fe65ecd70c0 1 bucket sync caught up with source: 2020-10-09T17:24:59.853 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:24:59.853 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:24:59.853 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:24:59.846+0000 7fe65ecd70c0 0 bucket checkpoint complete 2020-10-09T17:24:59.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:24:59.885 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:24:59.929 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:24:59.930 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:00.158 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:00.154+0000 7fd4f9dce0c0 1 bucket sync caught up with source: 2020-10-09T17:25:00.159 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.159 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.159 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:00.154+0000 7fd4f9dce0c0 0 bucket checkpoint complete 2020-10-09T17:25:00.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:00.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:00.191 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:00.283 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:00.471 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:00.514 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:00.510+0000 7fac2993d0c0 1 bucket sync caught up with source: 2020-10-09T17:25:00.515 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.515 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.515 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:00.510+0000 7fac2993d0c0 0 bucket checkpoint complete 2020-10-09T17:25:00.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:00.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:00.541 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:00.588 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:00.588 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:00.846 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:00.842+0000 7f13d0de30c0 1 bucket sync caught up with source: 2020-10-09T17:25:00.846 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.846 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:00.847 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:00.842+0000 7f13d0de30c0 0 bucket checkpoint complete 2020-10-09T17:25:00.875 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:25:00.893 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:00.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:00.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:00.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:01.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.203 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:25:01.204 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:25:01.215 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:01.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:01.274 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:01.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:01.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.519 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:25:01.520 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:25:01.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:01.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:01.585 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:01.595 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:01.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:01.823 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:25:01.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:01.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:01.846 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:01.891 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:01.892 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:02.094 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:25:02.128 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.129 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:25:02.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.131 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.131 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264301.840172_201.1", 2020-10-09T17:25:02.131 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:25:01.840172Z" 2020-10-09T17:25:02.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.131 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:25:02.132 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:25:02.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.134 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:02.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:02.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:02.429 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:25:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:25:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:25:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:25:02.429 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:02.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:25:02.431 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:02.431 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:02.431 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:02.431 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:02.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:02.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:02.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:25:02.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:02.433 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:02.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:25:02.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:02.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:02.434 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:02.434 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:25:02.434 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:02.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:02.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:02.435 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:02.436 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:25:02.437 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264301.840172_201.1", 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.840172Z", 2020-10-09T17:25:02.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:25:02.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:02.440 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:25:02.441 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:25:02.441 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:25:02.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.443 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:02.484 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:02.485 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:02.733 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:02.734 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:02.735 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:02.737 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.738 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:02.739 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:25:02.740 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264301.840172_201.1", 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:01.840172Z", 2020-10-09T17:25:02.741 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:25:02.742 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:25:02.743 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:02.743 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:25:02.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:25:02.744 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:25:02.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:02.745 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:02.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:02.787 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:03.057 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:25:03.057 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:25:03.057 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:25:03.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:25:03.057 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:25:03.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:03.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:03.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:03.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:03.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:25:03.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:03.060 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:03.060 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:03.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:03.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:03.061 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:25:03.061 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:03.062 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:03.062 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:25:03.062 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:03.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:03.063 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:03.063 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:25:03.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:03.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:03.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:03.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:25:03.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:03.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:03.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:03.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:03.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:03.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:03.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:03.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:03.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:03.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:03.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:25:03.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:03.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:03.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264301.840172_201.1", 2020-10-09T17:25:03.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:03.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:03.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:03.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.840172Z", 2020-10-09T17:25:03.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:03.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:25:03.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:03.072 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:25:03.072 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:25:03.072 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:25:03.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:03.072 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:25:03.073 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:25:03.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:03.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:03.124 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:03.174 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:03.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:03.418 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:03.414+0000 7f0cb824f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:03.418 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:25:03.419 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:08.422 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:08.418+0000 7f0cb824f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:08.422 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:25:08.423 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:13.426 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:13.422+0000 7f0cb824f0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:13.426 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.13.5, , ] 2020-10-09T17:25:13.426 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:18.429 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:18.425+0000 7f0cb824f0c0 1 bucket sync caught up with source: 2020-10-09T17:25:18.430 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:18.430 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:18.430 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:18.425+0000 7f0cb824f0c0 0 bucket checkpoint complete 2020-10-09T17:25:18.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:18.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:18.464 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:18.511 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:18.512 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:18.732 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:18.726+0000 7f847aa760c0 1 bucket sync caught up with source: 2020-10-09T17:25:18.732 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:18.732 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:18.733 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:18.726+0000 7f847aa760c0 0 bucket checkpoint complete 2020-10-09T17:25:18.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:18.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:18.760 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:18.812 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:18.812 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:19.028 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:19.025+0000 7fc6315530c0 1 bucket sync caught up with source: 2020-10-09T17:25:19.028 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:19.029 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.16.12, , ] 2020-10-09T17:25:19.029 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:19.025+0000 7fc6315530c0 0 bucket checkpoint complete 2020-10-09T17:25:19.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:19.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:19.074 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:19.122 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:19.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:19.363 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:19.358+0000 7fc97721e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:19.363 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:19.363 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:24.367 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:24.362+0000 7fc97721e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:24.367 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:24.367 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:29.371 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:29.366+0000 7fc97721e0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:29.371 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.14.5, , ] 2020-10-09T17:25:29.371 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:34.375 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:34.370+0000 7fc97721e0c0 1 bucket sync caught up with source: 2020-10-09T17:25:34.375 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:34.375 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:34.375 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:34.370+0000 7fc97721e0c0 0 bucket checkpoint complete 2020-10-09T17:25:34.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:34.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:34.409 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:34.458 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:34.459 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:34.692 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:34.686+0000 7efde059a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:34.739 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:34.740 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:39.696 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:39.690+0000 7efde059a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:39.696 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:39.697 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:44.701 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:44.694+0000 7efde059a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:44.701 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:44.701 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:49.703 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:49.698+0000 7efde059a0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:49.704 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.18.12, , ] 2020-10-09T17:25:49.704 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:54.708 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:54.702+0000 7efde059a0c0 1 bucket sync caught up with source: 2020-10-09T17:25:54.709 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:54.709 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:54.709 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:54.702+0000 7efde059a0c0 0 bucket checkpoint complete 2020-10-09T17:25:54.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:54.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:54.743 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:54.785 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:54.785 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:55.022 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:55.014+0000 7f7e099530c0 1 bucket sync caught up with source: 2020-10-09T17:25:55.048 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:55.048 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:55.049 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:55.014+0000 7f7e099530c0 0 bucket checkpoint complete 2020-10-09T17:25:55.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:55.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:55.052 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:55.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:55.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:55.336 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:55.333+0000 7fa4bb2660c0 1 bucket sync caught up with source: 2020-10-09T17:25:55.336 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:55.337 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:55.337 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:55.333+0000 7fa4bb2660c0 0 bucket checkpoint complete 2020-10-09T17:25:55.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:25:55.380 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:55.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:55.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.777 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:25:55.777 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:25:55.787 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:55.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.835 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:55.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:55.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:55.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:25:56.179 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:25:56.189 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:25:56.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:56.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:56.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:25:56.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:56.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:25:56.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:25:56.585 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:25:56.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:56.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:56.605 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:56.642 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:56.834 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:25:56.868 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:25:56.868 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:25:56.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264356.599102_209.1", 2020-10-09T17:25:56.870 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:25:56.599102Z" 2020-10-09T17:25:56.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:56.871 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:25:56.872 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:25:56.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:56.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:56.873 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:56.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:56.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:57.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:25:57.165 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:57.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:25:57.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:25:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264356.599102_209.1", 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.599102Z", 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.174 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:25:57.176 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:25:57.177 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:25:57.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:57.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:57.178 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:57.218 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:57.219 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:57.463 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:25:57.463 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:57.464 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:57.465 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:57.466 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:57.467 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:25:57.468 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:57.469 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:25:57.470 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264356.599102_209.1", 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:56.599102Z", 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.471 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:25:57.472 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:25:57.473 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:25:57.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:25:57.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:57.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:25:57.475 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:25:57.516 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:57.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:57.787 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:25:57.787 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:25:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:25:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:25:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:25:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264356.599102_209.1", 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.599102Z", 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.795 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:25:57.796 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:25:57.797 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:25:57.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:57.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:57.944 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:57.991 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:57.991 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:58.198 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:58.193+0000 7f52e62030c0 1 bucket sync caught up with source: 2020-10-09T17:25:58.199 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.199 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.199 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:58.193+0000 7f52e62030c0 0 bucket checkpoint complete 2020-10-09T17:25:58.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.231 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:58.514 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:58.510+0000 7f9d397900c0 1 bucket sync caught up with source: 2020-10-09T17:25:58.514 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.514 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.515 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:58.510+0000 7f9d397900c0 0 bucket checkpoint complete 2020-10-09T17:25:58.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.540 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:58.587 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:58.587 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:58.818 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:58.813+0000 7f77d65640c0 1 bucket sync caught up with source: 2020-10-09T17:25:58.818 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.819 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000005.20.17, , ] 2020-10-09T17:25:58.819 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:25:58.813+0000 7f77d65640c0 0 bucket checkpoint complete 2020-10-09T17:25:58.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:25:58.855 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:25:58.904 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:25:58.905 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:25:59.129 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:25:59.122+0000 7f3b49eb90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:25:59.129 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:25:59.129 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:04.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:04.126+0000 7f3b49eb90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:04.133 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:26:04.133 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:09.137 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:09.130+0000 7f3b49eb90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:09.137 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000004.20.9, , ] 2020-10-09T17:26:09.137 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:14.141 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:14.134+0000 7f3b49eb90c0 1 bucket sync caught up with source: 2020-10-09T17:26:14.142 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:14.142 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:14.142 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:14.134+0000 7f3b49eb90c0 0 bucket checkpoint complete 2020-10-09T17:26:14.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:14.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:14.175 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:14.222 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:14.222 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:14.459 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:14.454+0000 7fe31e6de0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:14.460 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:14.460 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:19.462 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:19.458+0000 7fe31e6de0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:19.462 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:19.463 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:24.467 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:24.462+0000 7fe31e6de0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:24.468 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:24.468 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:29.471 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:29.466+0000 7fe31e6de0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:29.471 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:29.471 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:34.475 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:34.470+0000 7fe31e6de0c0 1 waiting for incremental sync to catch up: 2020-10-09T17:26:34.476 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000006.25.17, , ] 2020-10-09T17:26:34.476 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:39.479 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:39.473+0000 7fe31e6de0c0 1 bucket sync caught up with source: 2020-10-09T17:26:39.480 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:39.480 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:39.480 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:39.473+0000 7fe31e6de0c0 0 bucket checkpoint complete 2020-10-09T17:26:39.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:39.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:39.512 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:39.600 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:39.823 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:39.817+0000 7f76ce8440c0 1 bucket sync caught up with source: 2020-10-09T17:26:39.845 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:39.845 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:39.846 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:39.817+0000 7f76ce8440c0 0 bucket checkpoint complete 2020-10-09T17:26:39.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:39.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:39.848 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:39.895 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:39.896 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:40.136 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:40.132+0000 7f52f78890c0 1 bucket sync caught up with source: 2020-10-09T17:26:40.136 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:40.136 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000007.27.14, , ] 2020-10-09T17:26:40.136 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:40.132+0000 7f52f78890c0 0 bucket checkpoint complete 2020-10-09T17:26:40.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:26:40.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:40.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.308 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:40.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.646 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone2, test-zone1} 2020-10-09T17:26:40.646 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:26:40.654 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:40.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.703 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:40.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:40.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:40.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone1} 2020-10-09T17:26:41.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:26:41.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:41.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:41.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:41.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:41.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:41.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:41.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:41.637 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-20 zones={test-zone3, test-zone2} 2020-10-09T17:26:41.638 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-10-09T17:26:41.638 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-21 2020-10-09T17:26:41.673 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-22 2020-10-09T17:26:41.739 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-23 2020-10-09T17:26:41.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:41.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:41.842 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:41.888 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:41.888 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:42.087 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:26:42.122 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:26:42.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:26:42.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264401.832028_239.1", 2020-10-09T17:26:42.124 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:26:41.832028Z" 2020-10-09T17:26:42.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.125 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:26:42.126 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:26:42.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.127 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:42.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:42.170 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:42.437 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:26:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:26:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:26:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:26:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:26:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:26:42.443 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264401.832028_239.1", 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.832028Z", 2020-10-09T17:26:42.444 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:26:42.445 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:26:42.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:42.446 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:26:42.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:26:42.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:26:42.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.448 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:42.492 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:42.492 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:42.972 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:26:42.972 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:26:42.972 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:26:42.972 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:26:42.972 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:26:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:26:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:42.974 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:26:42.975 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:42.975 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:42.975 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:42.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:42.976 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:26:42.977 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:42.977 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:42.977 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:42.977 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:26:42.978 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:42.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.979 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:26:42.979 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:42.979 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:42.980 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264401.832028_239.1", 2020-10-09T17:26:42.981 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:42.982 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:42.982 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.832028Z", 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:42.983 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:26:42.984 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:26:42.984 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:26:42.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:42.984 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:26:42.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:26:42.985 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:26:42.986 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:42.987 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:43.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:43.026 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:43.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:26:43.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:26:43.395 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:43.396 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:43.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:43.398 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:26:43.399 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:43.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264401.832028_239.1", 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:43.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.832028Z", 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:43.403 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:26:43.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:26:43.404 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:26:43.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:43.404 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:26:43.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:26:43.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:43.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:43.632 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:43.681 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:43.682 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:43.900 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:43.896+0000 7ff12af980c0 1 bucket sync caught up with source: 2020-10-09T17:26:43.900 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:43.900 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:43.900 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:43.896+0000 7ff12af980c0 0 bucket checkpoint complete 2020-10-09T17:26:43.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:43.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:43.941 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:43.995 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:44.178 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:44.173+0000 7f1976ede0c0 1 bucket sync caught up with source: 2020-10-09T17:26:44.179 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:44.179 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:44.179 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:44.173+0000 7f1976ede0c0 0 bucket checkpoint complete 2020-10-09T17:26:44.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.206 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:44.252 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:44.253 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:44.482 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:44.476+0000 7f86c9f5e0c0 1 bucket sync caught up with source: 2020-10-09T17:26:44.482 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:44.482 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.42.11, , ] 2020-10-09T17:26:44.482 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:44.476+0000 7f86c9f5e0c0 0 bucket checkpoint complete 2020-10-09T17:26:44.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.521 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:44.570 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:44.570 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:44.793 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:44.785+0000 7fb8ff8d20c0 1 bucket sync caught up with source: 2020-10-09T17:26:44.794 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.30.5, , ] 2020-10-09T17:26:44.794 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.30.5, , ] 2020-10-09T17:26:44.794 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:44.785+0000 7fb8ff8d20c0 0 bucket checkpoint complete 2020-10-09T17:26:44.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:44.828 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:44.874 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:44.875 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:45.113 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:45.105+0000 7fc6a0caf0c0 1 bucket sync caught up with source: 2020-10-09T17:26:45.114 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.30.5, , ] 2020-10-09T17:26:45.114 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.30.5, , ] 2020-10-09T17:26:45.114 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:45.105+0000 7fc6a0caf0c0 0 bucket checkpoint complete 2020-10-09T17:26:45.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:45.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:45.145 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:45.192 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:45.193 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:45.421+0000 7f95c63390c0 1 bucket sync caught up with source: 2020-10-09T17:26:45.428 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2020-10-09T17:26:45.428 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2020-10-09T17:26:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:45.421+0000 7f95c63390c0 0 bucket checkpoint complete 2020-10-09T17:26:45.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:45.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:45.456 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:45.503 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:45.504 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:45.754 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:45.748+0000 7fb685c640c0 1 bucket sync caught up with source: 2020-10-09T17:26:45.755 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.28.5, , ] 2020-10-09T17:26:45.755 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.28.5, , ] 2020-10-09T17:26:45.755 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:45.748+0000 7fb685c640c0 0 bucket checkpoint complete 2020-10-09T17:26:45.781 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-21 zones={test-zone2, test-zone1} 2020-10-09T17:26:45.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:45.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:45.840 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:45.840 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:45.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:45.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:45.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:45.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:45.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:45.988 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-21 zones={test-zone2, test-zone1} 2020-10-09T17:26:45.989 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-21 zones={test-zone3, test-zone1} 2020-10-09T17:26:45.998 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:46.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:46.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.054 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-21 zones={test-zone3, test-zone1} 2020-10-09T17:26:46.188 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-21 zones={test-zone3, test-zone2} 2020-10-09T17:26:46.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:46.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:46.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:46.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:46.392 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-21 zones={test-zone3, test-zone2} 2020-10-09T17:26:46.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.393 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:46.436 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:46.437 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:46.677 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:46.672+0000 7f8a888a00c0 1 bucket sync caught up with source: 2020-10-09T17:26:46.677 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:46.678 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:46.678 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:46.672+0000 7f8a888a00c0 0 bucket checkpoint complete 2020-10-09T17:26:46.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.710 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:46.753 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:46.970 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:46.965+0000 7fdd770430c0 1 bucket sync caught up with source: 2020-10-09T17:26:46.970 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:46.971 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:46.971 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:46.965+0000 7fdd770430c0 0 bucket checkpoint complete 2020-10-09T17:26:46.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:46.999 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:47.044 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:47.044 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:47.283 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:47.276+0000 7fe0dd3560c0 1 bucket sync caught up with source: 2020-10-09T17:26:47.284 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:47.284 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2020-10-09T17:26:47.284 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:47.276+0000 7fe0dd3560c0 0 bucket checkpoint complete 2020-10-09T17:26:47.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:47.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:47.411 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:47.459 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:47.460 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:47.699 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:47.693+0000 7f764cbd70c0 1 bucket sync caught up with source: 2020-10-09T17:26:47.700 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.26.11, , ] 2020-10-09T17:26:47.700 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.26.11, , ] 2020-10-09T17:26:47.700 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:47.693+0000 7f764cbd70c0 0 bucket checkpoint complete 2020-10-09T17:26:47.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:47.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:47.733 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:47.781 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:48.013 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:48.005+0000 7ff5b382d0c0 1 bucket sync caught up with source: 2020-10-09T17:26:48.013 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.26.11, , ] 2020-10-09T17:26:48.013 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.26.11, , ] 2020-10-09T17:26:48.014 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:48.005+0000 7ff5b382d0c0 0 bucket checkpoint complete 2020-10-09T17:26:48.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:48.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:48.046 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:48.092 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:48.093 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:48.325+0000 7ff4262f60c0 1 bucket sync caught up with source: 2020-10-09T17:26:48.333 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2020-10-09T17:26:48.333 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2020-10-09T17:26:48.333 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:48.325+0000 7ff4262f60c0 0 bucket checkpoint complete 2020-10-09T17:26:48.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:48.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:48.361 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:48.407 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:48.408 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:48.653 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:48.648+0000 7f68c03320c0 1 bucket sync caught up with source: 2020-10-09T17:26:48.654 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.21.5, , ] 2020-10-09T17:26:48.654 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.21.5, , ] 2020-10-09T17:26:48.654 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:48.648+0000 7f68c03320c0 0 bucket checkpoint complete 2020-10-09T17:26:48.686 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-22 zones={test-zone2, test-zone1} 2020-10-09T17:26:48.697 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:48.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:48.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:48.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:48.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:48.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-22 zones={test-zone2, test-zone1} 2020-10-09T17:26:48.898 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-22 zones={test-zone3, test-zone1} 2020-10-09T17:26:48.907 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:48.954 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.955 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:48.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:48.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:49.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.099 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-22 zones={test-zone3, test-zone1} 2020-10-09T17:26:49.099 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-22 zones={test-zone3, test-zone2} 2020-10-09T17:26:49.108 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:49.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:49.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:49.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:49.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:49.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:49.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:49.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-22 zones={test-zone3, test-zone2} 2020-10-09T17:26:49.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.311 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:49.357 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:49.358 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:49.604 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:49.600+0000 7fa6ac7510c0 1 bucket sync caught up with source: 2020-10-09T17:26:49.605 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:49.605 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:49.605 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:49.600+0000 7fa6ac7510c0 0 bucket checkpoint complete 2020-10-09T17:26:49.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.637 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:49.686 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:49.686 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:49.913+0000 7f5232bea0c0 1 bucket sync caught up with source: 2020-10-09T17:26:49.921 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:49.921 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:49.921 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:49.913+0000 7f5232bea0c0 0 bucket checkpoint complete 2020-10-09T17:26:49.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:49.948 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:49.997 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:49.998 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:50.216 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:50.212+0000 7f6d324900c0 1 bucket sync caught up with source: 2020-10-09T17:26:50.217 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:50.217 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000002.11.5, , ] 2020-10-09T17:26:50.217 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:50.212+0000 7f6d324900c0 0 bucket checkpoint complete 2020-10-09T17:26:50.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.268 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:50.315 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:50.549+0000 7ff4ae7c40c0 1 bucket sync caught up with source: 2020-10-09T17:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.18.5, , ] 2020-10-09T17:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.18.5, , ] 2020-10-09T17:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:50.549+0000 7ff4ae7c40c0 0 bucket checkpoint complete 2020-10-09T17:26:50.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.592 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:50.637 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:50.637 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:50.841+0000 7ff4888200c0 1 bucket sync caught up with source: 2020-10-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000002.18.5, , ] 2020-10-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000002.18.5, , ] 2020-10-09T17:26:50.850 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:50.841+0000 7ff4888200c0 0 bucket checkpoint complete 2020-10-09T17:26:50.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:50.883 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:50.978 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:50.978 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:51.224 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:51.217+0000 7fa6b53a20c0 1 bucket sync caught up with source: 2020-10-09T17:26:51.225 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2020-10-09T17:26:51.225 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2020-10-09T17:26:51.225 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:51.217+0000 7fa6b53a20c0 0 bucket checkpoint complete 2020-10-09T17:26:51.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:51.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:51.253 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:51.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:51.300 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:51.536 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:51.532+0000 7f42bbc940c0 1 bucket sync caught up with source: 2020-10-09T17:26:51.536 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.21.11, , ] 2020-10-09T17:26:51.536 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.21.11, , ] 2020-10-09T17:26:51.537 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:51.532+0000 7f42bbc940c0 0 bucket checkpoint complete 2020-10-09T17:26:51.567 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-23 zones={test-zone2, test-zone1} 2020-10-09T17:26:51.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:51.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:51.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.781 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-23 zones={test-zone2, test-zone1} 2020-10-09T17:26:51.782 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-23 zones={test-zone3, test-zone1} 2020-10-09T17:26:51.790 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:51.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:51.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:51.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:51.984 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-23 zones={test-zone3, test-zone1} 2020-10-09T17:26:51.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-23 zones={test-zone3, test-zone2} 2020-10-09T17:26:51.995 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:52.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:52.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:52.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:52.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:52.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:52.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:52.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:52.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-23 zones={test-zone3, test-zone2} 2020-10-09T17:26:52.199 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-10-09T17:26:52.199 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-24 2020-10-09T17:26:52.233 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-25 2020-10-09T17:26:52.288 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-26 2020-10-09T17:26:52.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:52.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:52.442 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:52.479 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:52.683 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:26:52.720 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:52.721 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:26:52.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264412.432960_277.1", 2020-10-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:26:52.432960Z" 2020-10-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:52.723 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:26:52.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:26:52.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:52.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:52.726 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:52.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:52.765 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:53.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:53.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.019 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264412.432960_277.1", 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.432960Z", 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:26:53.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:26:53.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:26:53.023 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:26:53.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:53.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:53.025 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:53.064 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:53.064 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:26:53.318 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:53.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:26:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:53.320 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:53.321 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:53.322 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:53.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:53.324 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:53.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264412.432960_277.1", 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:26:53.326 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.432960Z", 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:26:53.327 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:26:53.328 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:26:53.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:26:53.330 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:26:53.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:53.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:26:53.332 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:26:53.373 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:53.373 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:53.641 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:26:53.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264412.432960_277.1", 2020-10-09T17:26:53.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.432960Z", 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.649 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:26:53.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:26:53.651 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:26:53.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:53.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:53.888 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:53.935 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:53.936 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:54.151+0000 7f486b7010c0 1 bucket sync caught up with source: 2020-10-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.157 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:54.151+0000 7f486b7010c0 0 bucket checkpoint complete 2020-10-09T17:26:54.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.189 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:54.237 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:54.441+0000 7f7f1da190c0 1 bucket sync caught up with source: 2020-10-09T17:26:54.449 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.449 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:54.441+0000 7f7f1da190c0 0 bucket checkpoint complete 2020-10-09T17:26:54.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.478 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:54.524 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:54.524 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:54.773 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:54.767+0000 7f5830a220c0 1 bucket sync caught up with source: 2020-10-09T17:26:54.773 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.774 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.11.6, , ] 2020-10-09T17:26:54.774 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:54.767+0000 7f5830a220c0 0 bucket checkpoint complete 2020-10-09T17:26:54.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:54.897 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:54.943 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:55.185 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.177+0000 7fe7b3fc10c0 1 bucket sync caught up with source: 2020-10-09T17:26:55.186 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-10-09T17:26:55.186 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-10-09T17:26:55.186 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.177+0000 7fe7b3fc10c0 0 bucket checkpoint complete 2020-10-09T17:26:55.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.220 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:55.266 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:55.499 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.493+0000 7f9ce2ff60c0 1 bucket sync caught up with source: 2020-10-09T17:26:55.500 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.12.12, , ] 2020-10-09T17:26:55.500 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.12.12, , ] 2020-10-09T17:26:55.500 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.493+0000 7f9ce2ff60c0 0 bucket checkpoint complete 2020-10-09T17:26:55.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.533 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:55.619 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:55.620 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:55.856 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.849+0000 7f78a429f0c0 1 bucket sync caught up with source: 2020-10-09T17:26:55.856 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2020-10-09T17:26:55.856 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2020-10-09T17:26:55.857 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:55.849+0000 7f78a429f0c0 0 bucket checkpoint complete 2020-10-09T17:26:55.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:55.885 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:55.932 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:55.933 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:56.164 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:56.159+0000 7f0ad26200c0 1 bucket sync caught up with source: 2020-10-09T17:26:56.164 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.8.12, , ] 2020-10-09T17:26:56.164 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.8.12, , ] 2020-10-09T17:26:56.165 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:56.159+0000 7f0ad26200c0 0 bucket checkpoint complete 2020-10-09T17:26:56.193 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-24 zones={test-zone2, test-zone1} 2020-10-09T17:26:56.204 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:56.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.251 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:56.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:56.319 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-24 zones={test-zone2, test-zone1} 2020-10-09T17:26:56.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-24 zones={test-zone3, test-zone1} 2020-10-09T17:26:56.328 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:56.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:56.382 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:56.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-24 zones={test-zone3, test-zone1} 2020-10-09T17:26:56.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-24 zones={test-zone3, test-zone2} 2020-10-09T17:26:56.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:56.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:56.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:56.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:56.570 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-24 zones={test-zone3, test-zone2} 2020-10-09T17:26:56.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:56.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:56.572 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:56.618 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:56.618 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:56.856 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:56.851+0000 7f8c671360c0 1 bucket sync caught up with source: 2020-10-09T17:26:56.857 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:56.857 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:56.857 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:56.851+0000 7f8c671360c0 0 bucket checkpoint complete 2020-10-09T17:26:56.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:56.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:56.891 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:56.939 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:56.939 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:57.176 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:57.169+0000 7fae39dff0c0 1 bucket sync caught up with source: 2020-10-09T17:26:57.177 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:57.177 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:57.177 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:57.169+0000 7fae39dff0c0 0 bucket checkpoint complete 2020-10-09T17:26:57.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.206 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:57.252 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:57.252 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:57.482 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:57.475+0000 7f172582f0c0 1 bucket sync caught up with source: 2020-10-09T17:26:57.482 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:57.483 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.35.12, , ] 2020-10-09T17:26:57.483 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:57.475+0000 7f172582f0c0 0 bucket checkpoint complete 2020-10-09T17:26:57.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.619 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:57.669 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:57.669 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:57.899 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:57.893+0000 7fee0404f0c0 1 bucket sync caught up with source: 2020-10-09T17:26:57.899 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2020-10-09T17:26:57.899 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2020-10-09T17:26:57.899 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:57.893+0000 7fee0404f0c0 0 bucket checkpoint complete 2020-10-09T17:26:57.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:57.932 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:57.983 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:57.984 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:58.197 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:58.189+0000 7fbfa94780c0 1 bucket sync caught up with source: 2020-10-09T17:26:58.364 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.30.6, , ] 2020-10-09T17:26:58.364 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.30.6, , ] 2020-10-09T17:26:58.364 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:58.189+0000 7fbfa94780c0 0 bucket checkpoint complete 2020-10-09T17:26:58.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:58.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:58.366 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:58.414 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:58.415 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:58.653 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:58.645+0000 7f5f7ebca0c0 1 bucket sync caught up with source: 2020-10-09T17:26:58.653 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.39.12, , ] 2020-10-09T17:26:58.654 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.39.12, , ] 2020-10-09T17:26:58.654 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:58.645+0000 7f5f7ebca0c0 0 bucket checkpoint complete 2020-10-09T17:26:58.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:58.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:58.683 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:58.732 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:58.733 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:58.989 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:58.983+0000 7fcbc8dd50c0 1 bucket sync caught up with source: 2020-10-09T17:26:58.990 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.39.12, , ] 2020-10-09T17:26:58.990 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.39.12, , ] 2020-10-09T17:26:58.990 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:58.983+0000 7fcbc8dd50c0 0 bucket checkpoint complete 2020-10-09T17:26:59.017 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-25 zones={test-zone2, test-zone1} 2020-10-09T17:26:59.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:59.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.079 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:59.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:59.147 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-25 zones={test-zone2, test-zone1} 2020-10-09T17:26:59.147 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-25 zones={test-zone3, test-zone1} 2020-10-09T17:26:59.157 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:59.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.203 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:59.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:59.270 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-25 zones={test-zone3, test-zone1} 2020-10-09T17:26:59.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-25 zones={test-zone3, test-zone2} 2020-10-09T17:26:59.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:26:59.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.327 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:26:59.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:26:59.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:26:59.402 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-25 zones={test-zone3, test-zone2} 2020-10-09T17:26:59.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:59.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:59.404 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:59.446 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:59.447 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:59.692 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:59.687+0000 7f255910a0c0 1 bucket sync caught up with source: 2020-10-09T17:26:59.693 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:26:59.693 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:26:59.693 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:26:59.687+0000 7f255910a0c0 0 bucket checkpoint complete 2020-10-09T17:26:59.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:59.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:26:59.724 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:26:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:26:59.765 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:26:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:59.981+0000 7f66ce16b0c0 1 bucket sync caught up with source: 2020-10-09T17:26:59.987 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:26:59.987 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:26:59.987 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:26:59.981+0000 7f66ce16b0c0 0 bucket checkpoint complete 2020-10-09T17:27:00.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.013 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:27:00.059 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:00.060 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:00.296 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:00.291+0000 7f472fd850c0 1 bucket sync caught up with source: 2020-10-09T17:27:00.296 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:27:00.296 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000001.37.12, , ] 2020-10-09T17:27:00.297 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:00.291+0000 7f472fd850c0 0 bucket checkpoint complete 2020-10-09T17:27:00.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.428 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:27:00.475 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:00.475 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:00.714 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:00.709+0000 7f4bdeca20c0 1 bucket sync caught up with source: 2020-10-09T17:27:00.714 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-10-09T17:27:00.715 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-10-09T17:27:00.715 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:00.709+0000 7f4bdeca20c0 0 bucket checkpoint complete 2020-10-09T17:27:00.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:00.739 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:27:00.785 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:00.785 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:01.021 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:01.013+0000 7f42efbed0c0 1 bucket sync caught up with source: 2020-10-09T17:27:01.103 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000001.32.12, , ] 2020-10-09T17:27:01.103 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000001.32.12, , ] 2020-10-09T17:27:01.103 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:01.013+0000 7f42efbed0c0 0 bucket checkpoint complete 2020-10-09T17:27:01.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:01.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:01.105 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:27:01.192 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:01.193 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:01.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:01.413+0000 7f2e541f30c0 1 bucket sync caught up with source: 2020-10-09T17:27:01.420 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , , , , 00000000003.37.6, , ] 2020-10-09T17:27:01.420 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , , , , 00000000003.37.6, , ] 2020-10-09T17:27:01.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:27:01.413+0000 7f2e541f30c0 0 bucket checkpoint complete 2020-10-09T17:27:01.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:01.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:27:01.450 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:27:01.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:01.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:01.741 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:01.735+0000 7ff598f290c0 1 bucket sync caught up with source: 2020-10-09T17:27:01.741 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , , , , 00000000003.37.6, , ] 2020-10-09T17:27:01.741 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , , , , 00000000003.37.6, , ] 2020-10-09T17:27:01.741 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:01.735+0000 7ff598f290c0 0 bucket checkpoint complete 2020-10-09T17:27:01.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-26 zones={test-zone2, test-zone1} 2020-10-09T17:27:01.785 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:27:01.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:01.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:27:01.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:01.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:27:01.915 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-26 zones={test-zone2, test-zone1} 2020-10-09T17:27:01.916 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-26 zones={test-zone3, test-zone1} 2020-10-09T17:27:01.926 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:27:01.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:01.976 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:27:01.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:02.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:27:02.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-26 zones={test-zone3, test-zone1} 2020-10-09T17:27:02.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-26 zones={test-zone3, test-zone2} 2020-10-09T17:27:02.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:27:02.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:02.115 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:27:02.121 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-10-09T17:27:02.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-10-09T17:27:02.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-26 zones={test-zone3, test-zone2} 2020-10-09T17:27:02.186 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-10-09T17:27:02.186 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-27 2020-10-09T17:27:02.220 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-28 2020-10-09T17:27:02.278 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-29 2020-10-09T17:27:02.525 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-10-09T17:27:02.526 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-30 2020-10-09T17:27:02.566 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-31 2020-10-09T17:27:02.626 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-32 2020-10-09T17:27:02.890 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-10-09T17:27:02.891 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-33 2020-10-09T17:27:02.925 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-34 2020-10-09T17:27:02.996 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-35 2020-10-09T17:27:05.246 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-10-09T17:27:05.247 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-36 2020-10-09T17:27:05.282 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-37 2020-10-09T17:27:05.342 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-38 2020-10-09T17:27:05.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.405 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:05.440 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:05.441 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:05.635 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:27:05.662 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:27:05.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:05.663 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:05.663 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:27:05.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:05.664 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264425.387176_360.1", 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:05.387176Z" 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:05.665 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:27:05.666 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:27:05.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.668 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:05.708 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:05.708 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:05.944 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:05.944 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:27:05.944 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:27:05.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:27:05.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:27:05.946 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:05.947 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:05.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:05.949 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:05.950 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264425.387176_360.1", 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:05.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.387176Z", 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:27:05.952 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:27:05.953 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:27:05.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:05.953 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:05.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:27:05.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:27:05.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:05.955 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:06.249 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:27:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:27:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:27:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:27:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:27:06.250 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:06.251 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:06.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:27:06.253 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:06.254 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:06.255 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:06.256 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264425.387176_360.1", 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.387176Z", 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:06.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:06.258 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:27:06.260 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:27:06.260 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:27:06.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:06.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:06.262 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:06.301 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:06.301 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:06.569 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:27:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:27:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:27:06.570 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:27:06.571 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:06.572 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:06.573 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:06.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:06.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264425.387176_360.1", 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.387176Z", 2020-10-09T17:27:06.577 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:27:06.578 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:27:06.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:06.579 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:06.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:27:06.598 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: qMowmWdmCRhKc6bo{1}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.598 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: qMowmWdmCRhKc6bo{1}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.598 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: qMowmWdmCRhKc6bo{1}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.598 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: qMowmWdmCRhKc6bo{1}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.599 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:27:06.600 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.0, version: qMowmWdmCRhKc6bo{2}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=Lf9xEoA2ZBRpDuht}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: Lf9xEoA2ZBRpDuht,0=op: Op::set_head, part_num: 0, part_tag: Lf9xEoA2ZBRpDuht}} 2020-10-09T17:27:06.600 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.593+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: Lf9xEoA2ZBRpDuht 2020-10-09T17:27:06.603 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.597+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: Lf9xEoA2ZBRpDuht 2020-10-09T17:27:06.603 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.597+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:27:06.603 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.597+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:27:06.605 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-10-09T17:27:06.601+0000 7f3a7bed7700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.0, version: qMowmWdmCRhKc6bo{3}, oid_prefix: data_log.0.D8yGg12mb7X, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: Lf9xEoA2ZBRpDuht, tags: {{0=Lf9xEoA2ZBRpDuht}}, journal: {{{}} 2020-10-09T17:27:06.722 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.715+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: KfFiaNBqnPitBaxW{1}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.722 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.715+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: KfFiaNBqnPitBaxW{1}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.722 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.715+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: KfFiaNBqnPitBaxW{1}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.722 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.715+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: KfFiaNBqnPitBaxW{1}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.723 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.715+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:27:06.724 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.719+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.0, version: KfFiaNBqnPitBaxW{2}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=MP8NjY6iilACPLtr}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: MP8NjY6iilACPLtr,0=op: Op::set_head, part_num: 0, part_tag: MP8NjY6iilACPLtr}} 2020-10-09T17:27:06.724 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.719+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: MP8NjY6iilACPLtr 2020-10-09T17:27:06.726 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.719+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: MP8NjY6iilACPLtr 2020-10-09T17:27:06.727 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.719+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:27:06.727 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.719+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:27:06.729 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:27:06.723+0000 7fd7cbf77700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.0, version: KfFiaNBqnPitBaxW{3}, oid_prefix: data_log.0.QOGWW0I4Y/0, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: MP8NjY6iilACPLtr, tags: {{0=MP8NjY6iilACPLtr}}, journal: {{{}} 2020-10-09T17:27:06.823 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.817+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: ZtsTmXibrJkp7vsZ{1}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.824 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.817+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: ZtsTmXibrJkp7vsZ{1}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.824 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.817+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: ZtsTmXibrJkp7vsZ{1}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.824 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.817+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: ZtsTmXibrJkp7vsZ{1}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:27:06.824 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.817+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:27:06.826 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.821+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.0, version: ZtsTmXibrJkp7vsZ{2}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=Cpv9llUgFz41Skfp}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: Cpv9llUgFz41Skfp,0=op: Op::set_head, part_num: 0, part_tag: Cpv9llUgFz41Skfp}} 2020-10-09T17:27:06.826 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.821+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: Cpv9llUgFz41Skfp 2020-10-09T17:27:06.829 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.821+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: Cpv9llUgFz41Skfp 2020-10-09T17:27:06.829 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.821+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:27:06.829 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.821+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:27:06.831 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:06.825+0000 7f3f727bc700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.0, version: ZtsTmXibrJkp7vsZ{3}, oid_prefix: data_log.0.s2SpMq60as+, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: Cpv9llUgFz41Skfp, tags: {{0=Cpv9llUgFz41Skfp}}, journal: {{{}} 2020-10-09T17:27:06.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-10-09T17:27:06.936 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-39 2020-10-09T17:27:06.964 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-40 2020-10-09T17:27:07.019 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-41 2020-10-09T17:27:07.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.076 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:07.110 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:07.111 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:07.286 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=eb32ec64-e1b3-4311-8266-3310720f134e 2020-10-09T17:27:07.311 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:27:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:16:25.872885Z" 2020-10-09T17:27:07.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:27:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:27:07.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:07.314 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:18:37.851306Z" 2020-10-09T17:27:07.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.315 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264427.056998_374.1", 2020-10-09T17:27:07.315 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.056998Z" 2020-10-09T17:27:07.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.316 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:27:07.317 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:27:07.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.319 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:07.362 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:07.362 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:07.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:07.609 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:07.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:07.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:07.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:07.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:07.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264427.056998_374.1", 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:07.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.056998Z", 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:07.617 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:27:07.618 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:27:07.618 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:27:07.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:07.618 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:07.619 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:27:07.619 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:27:07.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.621 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:07.660 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:07.915 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:27:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:27:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:27:07.915 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:27:07.916 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:07.917 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:07.918 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:07.919 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:07.920 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:27:07.921 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264427.056998_374.1", 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:27:07.922 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.056998Z", 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:27:07.923 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:27:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:27:07.924 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:27:07.924 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:27:07.925 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:27:07.925 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:27:07.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:27:07.927 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:27:07.965 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:07.966 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:08.237 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:08.238 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:27:08.238 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:27:08.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:27:08.239 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:27:08.239 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:08.239 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:08.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:08.240 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:27:08.240 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:08.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:27:08.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:08.241 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:08.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:27:08.242 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:08.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:27:08.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:08.243 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:27:08.243 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:08.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:27:08.244 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:27:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:08.245 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:08.246 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:08.246 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:27:08.246 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:08.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:08.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:08.247 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:27:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:27:08.248 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:08.249 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:08.249 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:08.249 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:27:08.250 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:08.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:08.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:08.251 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:27:08.251 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:08.251 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:27:08.252 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264427.056998_374.1", 2020-10-09T17:27:08.252 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:27:08.252 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:27:08.253 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:27:08.253 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.056998Z", 2020-10-09T17:27:08.253 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:27:08.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.254 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.254 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:08.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:08.255 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:27:08.255 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:27:08.255 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:27:08.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:08.256 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:08.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:27:08.258 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-10-09T17:27:08.259 INFO:teuthology.orchestra.run:waiting for 300 2020-10-09T17:27:08.260 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:08.253+0000 7f40c563d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 19501) UID: 0 2020-10-09T17:27:08.260 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:27:08.253+0000 7f40dcda5100 -1 shutting down 2020-10-09T17:27:10.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-10-09T17:27:14.361 INFO:tasks.rgw.c1.client.1:Stopped 2020-10-09T17:27:14.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:27:14.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:27:14.363 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:27:14.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:14.402 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:14.426 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:14.419+0000 7f82361b70c0 0 NOTICE: overriding master zone: cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:27:14.427 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:14.419+0000 7f82361b70c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:27:14.427 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:14.419+0000 7f82361b70c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:27:14.429 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:14.423+0000 7f82361b70c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:27:14.429 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:27:14.423+0000 7f82361b70c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:27:14.440 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:27:14.441 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:27:14.442 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:27:14.443 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:27:14.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:27:14.445 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:27:14.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:14.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:27:14.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:27:14.447 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:27:14.526 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:27:14.527 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:27:14.835 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 1, 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "eb32ec64-e1b3-4311-8266-3310720f134e", 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602263785.872885_64.1", 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:27:14.836 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602263917.851306_97.1", 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264427.056998_374.1" 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.837 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.838 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:27:14.839 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:27:14.840 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:27:14.841 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:27:14.842 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:27:14.844 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:27:14.846 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.847 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:27:14.848 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:27:14.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:27:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:27:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.859 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:27:14.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:27:14.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:27:14.862 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.863 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:27:14.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:27:14.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:27:14.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:27:14.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 3 2020-10-09T17:27:14.865 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:27:14.866 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-10-09T17:27:16.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-10-09T17:28:14.867 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-42 2020-10-09T17:28:14.945 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-43 2020-10-09T17:28:14.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:28:14.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:28:14.992 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:28:15.028 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:28:15.028 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:28:15.243 INFO:teuthology.orchestra.run.smithi184.stderr:No --period given, using current period=f137cd21-eb16-49e0-9fe5-a2877c0fdad4 2020-10-09T17:28:15.277 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:28:15.277 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:28:15.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:28:15.277 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-10-09T17:28:15.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:28:15.278 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "0.000000" 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264494.984679_341.1", 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:28:14.984679Z" 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:28:15.279 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:28:15.280 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-10-09T17:28:15.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:28:15.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:28:15.282 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:28:15.321 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:28:15.321 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:28:15.584 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:28:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:28:15.584 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 3 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:28:15.585 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 80, 2020-10-09T17:28:15.586 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 3 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:28:15.587 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:28:15.588 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 3 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:28:15.589 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 3 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.590 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264494.984679_341.1", 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:28:15.591 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.984679Z", 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 3 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.592 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:28:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 80, 2020-10-09T17:28:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 80 2020-10-09T17:28:15.593 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.593 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:28:15.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-10-09T17:28:15.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:28:15.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:28:15.595 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:28:15.679 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:28:15.680 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:28:15.700 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:28:15.693+0000 7fdb6fe720c0 0 NOTICE: overriding master zone: bf2453fb-36ea-45e0-8abf-a48b7c4b1b84 2020-10-09T17:28:15.701 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:28:15.693+0000 7fdb6fe720c0 0 failed reading obj info from .rgw.root:zone_info.879c40c2-988f-403b-a66c-6906292d3ac2: (2) No such file or directory 2020-10-09T17:28:15.701 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:28:15.693+0000 7fdb6fe720c0 0 WARNING: could not read zone params for zone id=879c40c2-988f-403b-a66c-6906292d3ac2 name=test-zone4 2020-10-09T17:28:15.701 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:28:15.697+0000 7fdb6fe720c0 0 failed reading obj info from .rgw.root:zone_info.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84: (2) No such file or directory 2020-10-09T17:28:15.702 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:28:15.697+0000 7fdb6fe720c0 0 WARNING: could not read zone params for zone id=bf2453fb-36ea-45e0-8abf-a48b7c4b1b84 name=test-zone2 2020-10-09T17:28:15.710 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:28:15.710 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "test-zone1.rgw.meta:root", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "test-zone1.rgw.control", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "test-zone1.rgw.log", 2020-10-09T17:28:15.711 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2020-10-09T17:28:15.712 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "test-zone1.rgw.otp", 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:28:15.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2020-10-09T17:28:15.714 INFO:teuthology.orchestra.run.smithi060.stdout: "compression_type": "random" 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:15.715 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:28:15.716 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2020-10-09T17:28:15.716 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:28:15.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:28:15.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:28:15.718 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:28:15.799 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:28:15.799 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:28:16.105 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:28:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:28:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 1, 2020-10-09T17:28:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:28:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [ 2020-10-09T17:28:16.106 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:28:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:28:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:28:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:28:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.107 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:28:16.108 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:28:16.109 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:28:16.110 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:28:16.111 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:28:16.112 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone4", 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:28:16.113 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:28:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:28:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "pubsub", 2020-10-09T17:28:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "false", 2020-10-09T17:28:16.114 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [ 2020-10-09T17:28:16.115 INFO:teuthology.orchestra.run.smithi060.stdout: "test-zone1" 2020-10-09T17:28:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:28:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:28:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:28:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:28:16.118 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:28:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.119 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:28:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:28:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:28:16.120 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:28:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:28:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:28:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:28:16.122 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:28:16.123 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:28:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:28:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:28:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:28:16.124 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:28:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:28:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.125 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.126 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.128 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:28:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:28:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:28:16.132 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 530799950 2020-10-09T17:28:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:28:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1014881947 2020-10-09T17:28:16.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 2021681268 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:28:16.134 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:28:16.135 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:28:16.136 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:28:16.137 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:28:16.138 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:28:16.139 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-10-09T17:29:16.141 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-44 2020-10-09T17:29:16.188 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-45 2020-10-09T17:29:16.257 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-10-09T17:29:16.258 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-10-09T17:29:16.261 INFO:tasks.rgw.c1.client.1:Started 2020-10-09T17:29:16.261 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-10-09T17:29:16.275 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:29:16.276 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:29:16.279 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8001: Connection refused 2020-10-09T17:29:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-10-09T17:29:17.282 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-10-09T17:29:17.297 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-10-09T17:29:17.297 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-10-09T17:29:17.302 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-10-09T17:29:17.303 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-10-09T17:29:17.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.304 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:17.387 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:17.388 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:17.604 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:29:17.638 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:17.638 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:17.638 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:17.639 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:17.640 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264556.231925_156.1", 2020-10-09T17:29:17.641 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:16.231925Z" 2020-10-09T17:29:17.641 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:17.641 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:17.642 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:29:17.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.643 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:17.684 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:17.685 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:17.927 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:17.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:17.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:17.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:17.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:17.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:17.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264556.231925_156.1", 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.231925Z", 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:17.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:17.937 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:29:17.938 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:29:17.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:17.939 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:17.979 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:17.980 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:18.231 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:18.232 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:18.233 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:18.234 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:18.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:18.236 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:18.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:18.238 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:18.239 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:18.239 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:18.239 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:18.240 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:18.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:18.240 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264556.231925_156.1", 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:18.241 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.231925Z", 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:18.242 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:18.243 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:29:18.243 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:29:18.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:18.243 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:18.244 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:29:18.245 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:29:18.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:18.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:18.246 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:18.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:18.286 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:18.554 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:18.554 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:18.555 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:18.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:18.557 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:18.558 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:18.559 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:18.560 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:18.561 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264556.231925_156.1", 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.231925Z", 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:18.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.563 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.564 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:18.564 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:18.564 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:18.564 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:29:18.564 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:29:18.565 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:18.565 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:18.566 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:29:18.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone2, test-zone1} 2020-10-09T17:29:18.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:18.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:18.702 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone2, test-zone1} 2020-10-09T17:29:18.703 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone3, test-zone1} 2020-10-09T17:29:18.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:18.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:18.834 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone3, test-zone1} 2020-10-09T17:29:18.835 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone1, test-zone2} 2020-10-09T17:29:18.845 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:18.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:18.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone1, test-zone2} 2020-10-09T17:29:18.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone3, test-zone2} 2020-10-09T17:29:18.971 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.018 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone3, test-zone2} 2020-10-09T17:29:19.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone1, test-zone3} 2020-10-09T17:29:19.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.142 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone1, test-zone3} 2020-10-09T17:29:19.210 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-39 zones={test-zone2, test-zone3} 2020-10-09T17:29:19.221 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.334 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-39 zones={test-zone2, test-zone3} 2020-10-09T17:29:19.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone2, test-zone1} 2020-10-09T17:29:19.349 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone2, test-zone1} 2020-10-09T17:29:19.466 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone3, test-zone1} 2020-10-09T17:29:19.476 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.589 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone3, test-zone1} 2020-10-09T17:29:19.589 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone1, test-zone2} 2020-10-09T17:29:19.600 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.646 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone1, test-zone2} 2020-10-09T17:29:19.710 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone3, test-zone2} 2020-10-09T17:29:19.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.833 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone3, test-zone2} 2020-10-09T17:29:19.834 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone1, test-zone3} 2020-10-09T17:29:19.846 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:19.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:19.957 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone1, test-zone3} 2020-10-09T17:29:19.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-40 zones={test-zone2, test-zone3} 2020-10-09T17:29:19.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.014 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-40 zones={test-zone2, test-zone3} 2020-10-09T17:29:20.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone2, test-zone1} 2020-10-09T17:29:20.099 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.222 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone2, test-zone1} 2020-10-09T17:29:20.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone3, test-zone1} 2020-10-09T17:29:20.235 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.282 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone3, test-zone1} 2020-10-09T17:29:20.350 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone1, test-zone2} 2020-10-09T17:29:20.361 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone1, test-zone2} 2020-10-09T17:29:20.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone3, test-zone2} 2020-10-09T17:29:20.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.534 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.597 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone3, test-zone2} 2020-10-09T17:29:20.598 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone1, test-zone3} 2020-10-09T17:29:20.610 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:20.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:20.726 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone1, test-zone3} 2020-10-09T17:29:21.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-41 zones={test-zone2, test-zone3} 2020-10-09T17:29:21.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.298 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:21.363 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-41 zones={test-zone2, test-zone3} 2020-10-09T17:29:21.364 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone2, test-zone1} 2020-10-09T17:29:21.373 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.422 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:21.490 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone2, test-zone1} 2020-10-09T17:29:21.491 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone3, test-zone1} 2020-10-09T17:29:21.501 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.552 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:21.633 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone3, test-zone1} 2020-10-09T17:29:21.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone1, test-zone2} 2020-10-09T17:29:21.646 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.698 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:21.762 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone1, test-zone2} 2020-10-09T17:29:21.763 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone3, test-zone2} 2020-10-09T17:29:21.774 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.826 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:21.894 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone3, test-zone2} 2020-10-09T17:29:21.895 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone1, test-zone3} 2020-10-09T17:29:21.908 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:21.996 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone1, test-zone3} 2020-10-09T17:29:22.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-42 zones={test-zone2, test-zone3} 2020-10-09T17:29:22.070 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.184 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-42 zones={test-zone2, test-zone3} 2020-10-09T17:29:22.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone2, test-zone1} 2020-10-09T17:29:22.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.330 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone2, test-zone1} 2020-10-09T17:29:22.331 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone3, test-zone1} 2020-10-09T17:29:22.343 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone3, test-zone1} 2020-10-09T17:29:22.466 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone1, test-zone2} 2020-10-09T17:29:22.486 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.535 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.607 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone1, test-zone2} 2020-10-09T17:29:22.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone3, test-zone2} 2020-10-09T17:29:22.624 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.679 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.751 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone3, test-zone2} 2020-10-09T17:29:22.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone1, test-zone3} 2020-10-09T17:29:22.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:22.896 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone1, test-zone3} 2020-10-09T17:29:22.897 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-43 zones={test-zone2, test-zone3} 2020-10-09T17:29:22.912 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:22.966 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.063 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-43 zones={test-zone2, test-zone3} 2020-10-09T17:29:23.064 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone2, test-zone1} 2020-10-09T17:29:23.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.145 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.210 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone2, test-zone1} 2020-10-09T17:29:23.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone3, test-zone1} 2020-10-09T17:29:23.226 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.287 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.366 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone3, test-zone1} 2020-10-09T17:29:23.367 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone1, test-zone2} 2020-10-09T17:29:23.387 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.512 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone1, test-zone2} 2020-10-09T17:29:23.514 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone3, test-zone2} 2020-10-09T17:29:23.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone3, test-zone2} 2020-10-09T17:29:23.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone1, test-zone3} 2020-10-09T17:29:23.682 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.734 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone1, test-zone3} 2020-10-09T17:29:23.806 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-44 zones={test-zone2, test-zone3} 2020-10-09T17:29:23.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:23.873 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:23.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-44 zones={test-zone2, test-zone3} 2020-10-09T17:29:23.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone2, test-zone1} 2020-10-09T17:29:23.956 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.074 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone2, test-zone1} 2020-10-09T17:29:24.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone3, test-zone1} 2020-10-09T17:29:24.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.210 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone3, test-zone1} 2020-10-09T17:29:24.210 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone1, test-zone2} 2020-10-09T17:29:24.222 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.270 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone1, test-zone2} 2020-10-09T17:29:24.339 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone3, test-zone2} 2020-10-09T17:29:24.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.471 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone3, test-zone2} 2020-10-09T17:29:24.471 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone1, test-zone3} 2020-10-09T17:29:24.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.593 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone1, test-zone3} 2020-10-09T17:29:24.594 INFO:tasks.rgw_multi.zone_rados:comparing bucket=wypfiu-45 zones={test-zone2, test-zone3} 2020-10-09T17:29:24.605 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-10-09T17:29:24.654 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-10-09T17:29:24.714 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=wypfiu-45 zones={test-zone2, test-zone3} 2020-10-09T17:29:24.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:24.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:24.716 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:24.749 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:24.749 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:24.980 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263785.865052_62.1", 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "user", 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.865052749Z", 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.981 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:24.982 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.983 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263785.872885_64.1", 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "user", 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885001Z", 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.985 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.733887_49.1", 2020-10-09T17:29:24.986 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.733887684Z", 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.738653_53.1", 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.738653976Z", 2020-10-09T17:29:24.989 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:24.990 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.991 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.740167_54.1", 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1", 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.740167740Z", 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.992 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.993 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.743542_55.1", 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:24.994 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1", 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.743542118Z", 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.995 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.764831_56.1", 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.764831074Z", 2020-10-09T17:29:24.997 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:24.998 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:24.998 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:24.998 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:24.999 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.769319_57.1", 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.000 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.769319759Z", 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.001 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.002 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.771065_58.1", 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2", 2020-10-09T17:29:25.003 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.771065035Z", 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.004 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.774595_59.1", 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2", 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.774595327Z", 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.006 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.008 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.123803_60.1", 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.123803579Z", 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.009 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.010 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.127933_61.1", 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.011 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.127933618Z", 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.129769_62.1", 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3", 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.129769137Z", 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.016 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.133520_63.1", 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3", 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.133520501Z", 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.017 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.018 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.356884_64.1", 2020-10-09T17:29:25.019 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.356884810Z", 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.020 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.021 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.361606_66.1", 2020-10-09T17:29:25.022 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.361606375Z", 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.023 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.363464_67.1", 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4", 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.363464549Z", 2020-10-09T17:29:25.025 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:25.026 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.027 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.367510_68.1", 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4", 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.367510128Z", 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.028 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.029 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.029 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:25.029 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.030 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.386611_69.1", 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.386611433Z", 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.031 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.032 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.391360_70.1", 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:25.033 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.391360027Z", 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.034 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.393032_71.1", 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5", 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.393032564Z", 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.036 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.397175_72.1", 2020-10-09T17:29:25.038 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5", 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.397175593Z", 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.040 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.440554_73.1", 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.440554181Z", 2020-10-09T17:29:25.041 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:25.042 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.043 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.444647_74.1", 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.444647257Z", 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.446535_75.1", 2020-10-09T17:29:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6", 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.446535350Z", 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.450323_76.1", 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6", 2020-10-09T17:29:25.049 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.450323041Z", 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.289813_77.1", 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.289813313Z", 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.052 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.053 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.294257_80.1", 2020-10-09T17:29:25.054 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.294257706Z", 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.296137_81.1", 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.296137549Z", 2020-10-09T17:29:25.057 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.058 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.059 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.300098_82.1", 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.300098832Z", 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.060 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.061 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.318125_83.1", 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.062 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.318125513Z", 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.064 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.064 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:25.064 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.064 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.065 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.322838_84.1", 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.322838970Z", 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.324748_85.1", 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.068 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.324748391Z", 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.069 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.328548_86.1", 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.328548439Z", 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.072 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.072 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.072 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.374771_87.1", 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.374771908Z", 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.074 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.379192_88.1", 2020-10-09T17:29:25.076 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.379192835Z", 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.078 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.380951_89.1", 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.079 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.380951565Z", 2020-10-09T17:29:25.080 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.080 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.080 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.080 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.384719_90.1", 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.082 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.384719553Z", 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.083 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.673745_91.1", 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.673745199Z", 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.085 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.086 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.678989_93.1", 2020-10-09T17:29:25.087 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.088 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.088 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.678989138Z", 2020-10-09T17:29:25.088 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.088 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.090 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.091 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.737348_94.1", 2020-10-09T17:29:25.092 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.737348546Z", 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.093 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.094 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.744698_95.1", 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.095 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.744698724Z", 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.097 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.097 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.097 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.097 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.844022_96.1", 2020-10-09T17:29:25.098 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.844022609Z", 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:25.100 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.851306_97.1", 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.101 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306494Z", 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.102 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.104 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.104 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.192132_44.1", 2020-10-09T17:29:25.104 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.104 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:25.105 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.192132424Z", 2020-10-09T17:29:25.105 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.105 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.105 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.105 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.106 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.196027_61.1", 2020-10-09T17:29:25.107 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.196027568Z", 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.108 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.109 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.197885_62.1", 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10", 2020-10-09T17:29:25.110 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.197885358Z", 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.111 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.201690_63.1", 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10", 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.201690295Z", 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.113 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.114 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.114 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.114 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.115 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.224925_64.1", 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.224925356Z", 2020-10-09T17:29:25.116 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:25.117 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.229526_65.1", 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.229526856Z", 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.119 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.120 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.231376_66.1", 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.121 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11", 2020-10-09T17:29:25.122 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.231376014Z", 2020-10-09T17:29:25.122 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.122 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.122 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.123 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.123 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.123 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.123 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:25.123 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.235568_67.1", 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11", 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.235568177Z", 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.126 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.288647_68.1", 2020-10-09T17:29:25.127 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.288647392Z", 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.130 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.130 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.292922_69.1", 2020-10-09T17:29:25.130 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.292922614Z", 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.131 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.132 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.133 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.133 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.295857_70.1", 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12", 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.295857593Z", 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.134 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.135 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.301783_71.1", 2020-10-09T17:29:25.136 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12", 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.301783945Z", 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.138 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.138 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:25.138 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.139 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.394203_80.1", 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.394203930Z", 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.140 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.141 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.398851_85.1", 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:25.143 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.398851813Z", 2020-10-09T17:29:25.143 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.143 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.143 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.143 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.144 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.401101_86.1", 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.145 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13", 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.401101583Z", 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.147 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.147 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:25.147 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.147 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.405020_87.1", 2020-10-09T17:29:25.148 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13", 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.405020770Z", 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.149 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.150 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.426024_88.1", 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.426024492Z", 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.151 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.152 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.431035_89.1", 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:25.154 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.431035284Z", 2020-10-09T17:29:25.155 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.155 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.155 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.155 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.156 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.433179_90.1", 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14", 2020-10-09T17:29:25.157 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.433179868Z", 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.437532_91.1", 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14", 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.437532851Z", 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.160 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.161 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.162 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.162 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.162 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.485384_92.1", 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.485384682Z", 2020-10-09T17:29:25.163 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.489772_93.1", 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.489772073Z", 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.166 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.168 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.491502_94.1", 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15", 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.491502823Z", 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.169 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.495381_95.1", 2020-10-09T17:29:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15", 2020-10-09T17:29:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.495381334Z", 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.529296_96.1", 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.529296713Z", 2020-10-09T17:29:25.175 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.176 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.177 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.533107_107.1", 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.533107387Z", 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.178 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.179 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.180 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.534931_108.1", 2020-10-09T17:29:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16", 2020-10-09T17:29:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.534931376Z", 2020-10-09T17:29:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.538686_109.1", 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16", 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.538686724Z", 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.184 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.186 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.560642_110.1", 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.560642819Z", 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.188 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.189 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.565909_111.1", 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.565909467Z", 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.190 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.191 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.568371_112.1", 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17", 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.568371015Z", 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.193 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:25.194 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.572591_113.1", 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17", 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.572591972Z", 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.196 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.616842_114.1", 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.616842253Z", 2020-10-09T17:29:25.199 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.200 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.620605_115.1", 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.620605579Z", 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.202 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.203 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.622618_116.1", 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.204 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18", 2020-10-09T17:29:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.622618991Z", 2020-10-09T17:29:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.205 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.206 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.626418_117.1", 2020-10-09T17:29:25.207 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18", 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.626418341Z", 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.645966_118.1", 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:25.210 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.645966499Z", 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.211 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.651155_119.1", 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.651155609Z", 2020-10-09T17:29:25.213 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.214 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.215 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.655670_120.1", 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.655670493Z", 2020-10-09T17:29:25.216 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.217 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.218 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.661706_121.1", 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.661706240Z", 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.219 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.220 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.673791_122.1", 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.221 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:25.222 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.673791703Z", 2020-10-09T17:29:25.222 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.222 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.222 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.223 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.681384_123.1", 2020-10-09T17:29:25.224 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.681384931Z", 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.226 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.974811_154.1", 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:25.227 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.974811077Z", 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.230 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.978899_165.1", 2020-10-09T17:29:25.230 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.230 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.978899378Z", 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.980746_166.1", 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19", 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.980746411Z", 2020-10-09T17:29:25.233 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:25.234 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.235 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.984814_167.1", 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19", 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.984814622Z", 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.236 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.990937_168.1", 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.238 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:25.239 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.990937161Z", 2020-10-09T17:29:25.239 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.239 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.239 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.239 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.240 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.996213_169.1", 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.241 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.996213796Z", 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.242 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.243 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.295175_170.1", 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.295175489Z", 2020-10-09T17:29:25.244 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.246 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.299816_191.1", 2020-10-09T17:29:25.247 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.247 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.247 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.299816303Z", 2020-10-09T17:29:25.247 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.301833_192.1", 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20", 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.301833065Z", 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.250 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.252 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.305660_193.1", 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20", 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.305660950Z", 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.253 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.254 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264301.829073_194.1", 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:01.829073153Z", 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.256 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.257 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264301.840172_201.1", 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.258 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:01.840172924Z", 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.259 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264356.587328_202.1", 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:56.587328671Z", 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.261 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.262 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264356.599102_209.1", 2020-10-09T17:29:25.263 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:56.599102815Z", 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.657350_220.1", 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.657350424Z", 2020-10-09T17:29:25.266 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.267 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.268 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.662094_223.1", 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.662094551Z", 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.269 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.270 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.664114_224.1", 2020-10-09T17:29:25.271 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21", 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.664114737Z", 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.272 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.273 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.668177_225.1", 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21", 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.668177389Z", 2020-10-09T17:29:25.274 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.694791_226.1", 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.694791618Z", 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.277 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.278 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.705509_227.1", 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.279 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.705509786Z", 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.280 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.281 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.707607_228.1", 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22", 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.707607520Z", 2020-10-09T17:29:25.282 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:25.283 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.711488_229.1", 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22", 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.711488882Z", 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.285 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.286 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.765125_230.1", 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.287 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.765125952Z", 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.769697_231.1", 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.769697460Z", 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.290 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.291 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.771452_232.1", 2020-10-09T17:29:25.292 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23", 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.771452268Z", 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.775560_233.1", 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23", 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.775560119Z", 2020-10-09T17:29:25.295 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:25.296 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.297 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.797874_234.1", 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.797874711Z", 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.298 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.299 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.805041_235.1", 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.300 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.805041511Z", 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.301 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.302 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.809433_236.1", 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.809433778Z", 2020-10-09T17:29:25.303 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.305 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.813654_237.1", 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.813654996Z", 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.307 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.825509_238.1", 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.308 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.825509079Z", 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.309 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.832028_239.1", 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:25.311 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.832028783Z", 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.312 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.216367_240.1", 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.216367795Z", 2020-10-09T17:29:25.314 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.315 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.316 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.221041_255.1", 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.221041216Z", 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.317 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.318 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.222904_256.1", 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.319 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24", 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.222904032Z", 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.320 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.227259_257.1", 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24", 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.227259403Z", 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.322 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.324 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.252949_258.1", 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.252949731Z", 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.257345_259.1", 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.327 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.257345653Z", 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.328 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.329 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.259289_260.1", 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25", 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.259289030Z", 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.330 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.331 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.262905_261.1", 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.332 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25", 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.262905010Z", 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.333 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.307988_262.1", 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.307988531Z", 2020-10-09T17:29:25.335 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.337 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.312877_263.1", 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.312877842Z", 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.338 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.339 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.315071_264.1", 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.340 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26", 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.315071641Z", 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.341 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.342 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.319616_265.1", 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26", 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.319616392Z", 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.343 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.345 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.341664_266.1", 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.341664786Z", 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.346 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.347 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.347803_267.1", 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.348 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.347803294Z", 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.349 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.351673_268.1", 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.351673675Z", 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.351 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.353 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.360647_269.1", 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.360647470Z", 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.354 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.355 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.370574_270.1", 2020-10-09T17:29:25.356 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.370574457Z", 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.357 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.358 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.375594_271.1", 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.375594201Z", 2020-10-09T17:29:25.359 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.361 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.391493_272.1", 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.391493979Z", 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.362 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.363 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.399765_273.1", 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.364 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.399765168Z", 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.365 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.415879_274.1", 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.415879375Z", 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.367 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.368 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.420739_275.1", 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.369 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.420739980Z", 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.370 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.371 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.427787_276.1", 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.427787757Z", 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.372 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.373 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.374 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.432960_277.1", 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.432960647Z", 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.375 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.376 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.204746_283.1", 2020-10-09T17:29:25.377 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.204746328Z", 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.378 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.379 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.209341_296.1", 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.209341959Z", 2020-10-09T17:29:25.380 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.382 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.211337_297.1", 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27", 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.211337050Z", 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.384 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.215528_298.1", 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27", 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.215528132Z", 2020-10-09T17:29:25.385 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.242661_299.1", 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.242661766Z", 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.388 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.389 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.248232_300.1", 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.390 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.248232520Z", 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.391 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.392 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.250316_301.1", 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28", 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.250316394Z", 2020-10-09T17:29:25.393 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.395 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.254254_302.1", 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28", 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.254254294Z", 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.396 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.397 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.298996_303.1", 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.398 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.298996399Z", 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.399 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.303839_304.1", 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.303839237Z", 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.401 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.402 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.305882_305.1", 2020-10-09T17:29:25.403 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29", 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.305882506Z", 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.405 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.309980_306.1", 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29", 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.309980653Z", 2020-10-09T17:29:25.406 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.408 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.338582_307.1", 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.338582404Z", 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.409 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.410 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.344508_308.1", 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:25.411 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.344508600Z", 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.414 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.393075_309.1", 2020-10-09T17:29:25.414 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.414 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:25.414 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.393075237Z", 2020-10-09T17:29:25.415 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.415 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.415 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.415 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.416 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.416 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.416 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.416 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.399422_310.1", 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.417 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.399422139Z", 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.418 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.419 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.458912_311.1", 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.458912136Z", 2020-10-09T17:29:25.420 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.421 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.422 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.467949_312.1", 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.467949620Z", 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.423 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.424 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.547125_313.1", 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:25.425 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.547125552Z", 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.426 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.552027_314.1", 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.552027100Z", 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.428 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.429 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.554041_315.1", 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.430 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30", 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.554041904Z", 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.431 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.432 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.558805_316.1", 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30", 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.558805398Z", 2020-10-09T17:29:25.433 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:25.434 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.435 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.584159_317.1", 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.584159383Z", 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.436 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.437 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.588880_318.1", 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:25.438 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.588880019Z", 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.439 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.590910_319.1", 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31", 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.590910718Z", 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.441 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.443 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.594932_320.1", 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31", 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.594932708Z", 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.445 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.649758_321.1", 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.649758180Z", 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.447 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.654484_322.1", 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.654484032Z", 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.449 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.450 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.656473_323.1", 2020-10-09T17:29:25.451 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32", 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.656473591Z", 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.453 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.660761_324.1", 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32", 2020-10-09T17:29:25.454 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.660761391Z", 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.455 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.686330_325.1", 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.686330593Z", 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.457 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.458 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.692341_326.1", 2020-10-09T17:29:25.459 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.692341773Z", 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.460 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.461 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.744293_327.1", 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.744293216Z", 2020-10-09T17:29:25.462 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.463 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.464 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.755410_328.1", 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.755410652Z", 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.466 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.467 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.827943_329.1", 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.827943489Z", 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.468 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.469 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.832753_330.1", 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.470 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.832753298Z", 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.471 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.472 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.909951_331.1", 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.909951412Z", 2020-10-09T17:29:25.473 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.474 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.475 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.914593_332.1", 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.914593130Z", 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.476 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.477 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.916589_333.1", 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.478 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33", 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.916589104Z", 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.479 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.920637_334.1", 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33", 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.920637835Z", 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.481 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.483 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.950486_335.1", 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.950486877Z", 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.485 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.956282_336.1", 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.486 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.956282207Z", 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.487 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.488 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.958961_337.1", 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34", 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.958961739Z", 2020-10-09T17:29:25.489 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.490 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.490 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.490 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.490 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.490 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.491 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.967075_338.1", 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34", 2020-10-09T17:29:25.492 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.967075872Z", 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.493 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.494 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.018870_339.1", 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:25.495 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.018870930Z", 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.496 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.497 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.022759_340.1", 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.022759559Z", 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.498 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.499 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.024635_341.1", 2020-10-09T17:29:25.500 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35", 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.024635168Z", 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.502 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.028418_342.1", 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35", 2020-10-09T17:29:25.503 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.028418911Z", 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.504 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.048923_343.1", 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.048923618Z", 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.506 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.507 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.055329_344.1", 2020-10-09T17:29:25.508 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.055329261Z", 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.509 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.510 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264424.106351_345.1", 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:04.106351971Z", 2020-10-09T17:29:25.511 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.512 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264424.117605_346.1", 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:04.117605225Z", 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.514 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.515 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.177021_347.1", 2020-10-09T17:29:25.516 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.177021241Z", 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.517 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.518 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.189008_348.1", 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.189008779Z", 2020-10-09T17:29:25.519 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:25.520 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.521 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.266549_349.1", 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.266549024Z", 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.522 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.523 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.271211_350.1", 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:25.524 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.271211187Z", 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.273170_351.1", 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36", 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.273170349Z", 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.527 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.277147_352.1", 2020-10-09T17:29:25.529 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36", 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.277147722Z", 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.530 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.531 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.306602_353.1", 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.306602896Z", 2020-10-09T17:29:25.532 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:25.533 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.534 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.311387_354.1", 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.311387019Z", 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.535 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.536 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.313355_355.1", 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.537 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37", 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.313355999Z", 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.538 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.539 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.317179_356.1", 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37", 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.317179014Z", 2020-10-09T17:29:25.540 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:25.541 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.365415_357.1", 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.365415525Z", 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.543 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.544 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.371789_358.1", 2020-10-09T17:29:25.545 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.371789680Z", 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.547 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.374833_359.1", 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38", 2020-10-09T17:29:25.548 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.374833790Z", 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:25.549 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.387176_360.1", 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38", 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.387176842Z", 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.552 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.949266_361.1", 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.553 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.949266551Z", 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.554 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.953535_364.1", 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.953535193Z", 2020-10-09T17:29:25.556 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:25.557 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.558 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.955454_365.1", 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39", 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.955454424Z", 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.559 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.560 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.959356_366.1", 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39", 2020-10-09T17:29:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.959356816Z", 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.562 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.983754_367.1", 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.983754988Z", 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.564 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.565 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.988786_368.1", 2020-10-09T17:29:25.566 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.988786029Z", 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.568 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.991106_369.1", 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40", 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.991106852Z", 2020-10-09T17:29:25.569 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.570 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.571 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.995242_370.1", 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40", 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.995242709Z", 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.573 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.041964_371.1", 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:25.574 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.041964955Z", 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:25.575 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.576 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.048895_372.1", 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.048895642Z", 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.577 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.578 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.051484_373.1", 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.579 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41", 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.051484910Z", 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.580 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.056998_374.1", 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41", 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.056998203Z", 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:25.582 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:25.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:25.584 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:25.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:25.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:25.587 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:25.625 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:25.625 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:25.861 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:25.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.861 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263785.865052_62.1", 2020-10-09T17:29:25.861 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.865052749Z", 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.862 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263785.872885_64.1", 2020-10-09T17:29:25.864 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-10-09T17:29:25.865 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:25.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885001Z", 2020-10-09T17:29:25.865 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.865 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.866 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.866 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.866 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.733887_49.1", 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:25.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.733887684Z", 2020-10-09T17:29:25.869 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.869 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.869 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.870 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.870 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:25.870 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.871 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.871 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.738653_53.1", 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.738653976Z", 2020-10-09T17:29:25.872 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.873 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.740167_54.1", 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1", 2020-10-09T17:29:25.875 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.740167740Z", 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.876 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.877 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.878 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.878 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.878 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.743542_55.1", 2020-10-09T17:29:25.880 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.880 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1", 2020-10-09T17:29:25.880 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.743542118Z", 2020-10-09T17:29:25.880 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.880 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.882 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.882 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.882 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.883 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.764831_56.1", 2020-10-09T17:29:25.883 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.884 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:25.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.764831074Z", 2020-10-09T17:29:25.884 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.884 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.885 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.886 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.886 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:25.886 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.887 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.887 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.888 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.769319_57.1", 2020-10-09T17:29:25.889 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.889 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:25.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.769319759Z", 2020-10-09T17:29:25.890 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.890 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.890 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.891 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.891 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:25.892 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.892 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.892 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.893 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.894 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.771065_58.1", 2020-10-09T17:29:25.894 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.894 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2", 2020-10-09T17:29:25.895 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.771065035Z", 2020-10-09T17:29:25.895 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.895 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.895 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.896 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.896 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.896 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:25.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.897 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.897 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.899 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.774595_59.1", 2020-10-09T17:29:25.899 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.899 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2", 2020-10-09T17:29:25.899 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.774595327Z", 2020-10-09T17:29:25.900 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.900 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.900 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.900 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.901 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.901 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:25.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.123803_60.1", 2020-10-09T17:29:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.123803579Z", 2020-10-09T17:29:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.127933_61.1", 2020-10-09T17:29:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.127933618Z", 2020-10-09T17:29:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.912 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.912 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.913 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.129769_62.1", 2020-10-09T17:29:25.913 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.914 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3", 2020-10-09T17:29:25.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.129769137Z", 2020-10-09T17:29:25.914 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.914 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.915 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.915 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.916 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:25.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.916 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.917 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.918 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.133520_63.1", 2020-10-09T17:29:25.918 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.919 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3", 2020-10-09T17:29:25.919 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.133520501Z", 2020-10-09T17:29:25.919 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.919 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.920 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.921 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:25.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.356884_64.1", 2020-10-09T17:29:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.356884810Z", 2020-10-09T17:29:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.361606_66.1", 2020-10-09T17:29:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.361606375Z", 2020-10-09T17:29:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.363464_67.1", 2020-10-09T17:29:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4", 2020-10-09T17:29:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.363464549Z", 2020-10-09T17:29:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.936 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.936 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.367510_68.1", 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4", 2020-10-09T17:29:25.937 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.367510128Z", 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.938 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.386611_69.1", 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.386611433Z", 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.940 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:25.941 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.391360_70.1", 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.391360027Z", 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.943 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.393032_71.1", 2020-10-09T17:29:25.945 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5", 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.393032564Z", 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.946 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.947 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.397175_72.1", 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.948 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5", 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.397175593Z", 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.949 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.440554_73.1", 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.440554181Z", 2020-10-09T17:29:25.951 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:25.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.444647_74.1", 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.444647257Z", 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.954 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.955 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.446535_75.1", 2020-10-09T17:29:25.956 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6", 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.446535350Z", 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.958 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.450323_76.1", 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.959 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6", 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323041Z", 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.960 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.289813_77.1", 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.289813313Z", 2020-10-09T17:29:25.962 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:25.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.294257_80.1", 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.294257706Z", 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.965 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.967 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.296137_81.1", 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.296137549Z", 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.968 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.969 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.300098_82.1", 2020-10-09T17:29:25.970 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.300098832Z", 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.318125_83.1", 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:25.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.318125513Z", 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.974 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.322838_84.1", 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.322838970Z", 2020-10-09T17:29:25.976 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:25.977 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.324748_85.1", 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.324748391Z", 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.979 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.980 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.328548_86.1", 2020-10-09T17:29:25.981 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.328548439Z", 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.983 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.374771_87.1", 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:25.984 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.374771908Z", 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.985 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.379192_88.1", 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.379192835Z", 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.987 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.989 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.380951_89.1", 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.380951565Z", 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.384719_90.1", 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.992 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.384719553Z", 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.993 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.673745_91.1", 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.673745199Z", 2020-10-09T17:29:25.995 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.678989_93.1", 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.678989138Z", 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:25.998 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:25.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.737348_94.1", 2020-10-09T17:29:26.002 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.737348546Z", 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.003 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.744698_95.1", 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.005 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.744698724Z", 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.844022_96.1", 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.844022609Z", 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.851306_97.1", 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306494Z", 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.192132_44.1", 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.192132424Z", 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.196027_61.1", 2020-10-09T17:29:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.196027568Z", 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.197885_62.1", 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10", 2020-10-09T17:29:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.197885358Z", 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.201690_63.1", 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10", 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.201690295Z", 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.022 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.224925_64.1", 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.224925356Z", 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.025 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.229526_65.1", 2020-10-09T17:29:26.027 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.229526856Z", 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.231376_66.1", 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11", 2020-10-09T17:29:26.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.231376014Z", 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.031 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.235568_67.1", 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11", 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.235568177Z", 2020-10-09T17:29:26.033 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:26.034 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.288647_68.1", 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.288647392Z", 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.036 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.038 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.292922_69.1", 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.292922614Z", 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.295857_70.1", 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.041 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12", 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.295857593Z", 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.042 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.301783_71.1", 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12", 2020-10-09T17:29:26.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.301783945Z", 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.045 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.394203_80.1", 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.394203930Z", 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.047 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.049 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.398851_85.1", 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.398851813Z", 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.050 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.051 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.401101_86.1", 2020-10-09T17:29:26.052 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13", 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.401101583Z", 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.054 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.405020_87.1", 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13", 2020-10-09T17:29:26.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.405020770Z", 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.056 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.426024_88.1", 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.426024492Z", 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.058 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.060 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.431035_89.1", 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.431035284Z", 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.061 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.062 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.433179_90.1", 2020-10-09T17:29:26.063 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14", 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.433179868Z", 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.437532_91.1", 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14", 2020-10-09T17:29:26.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.437532851Z", 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.067 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.485384_92.1", 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.485384682Z", 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.069 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:26.070 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.489772_93.1", 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.489772073Z", 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.072 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.073 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.491502_94.1", 2020-10-09T17:29:26.074 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15", 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.491502823Z", 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.495381_95.1", 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15", 2020-10-09T17:29:26.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.495381334Z", 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.529296_96.1", 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.529296713Z", 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.080 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.533107_107.1", 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.533107387Z", 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.083 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.084 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.534931_108.1", 2020-10-09T17:29:26.085 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16", 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.534931376Z", 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.538686_109.1", 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.088 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16", 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.538686724Z", 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.089 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.560642_110.1", 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.560642819Z", 2020-10-09T17:29:26.091 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.092 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.565909_111.1", 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.565909467Z", 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.094 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.096 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.568371_112.1", 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17", 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.568371015Z", 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.097 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.098 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.572591_113.1", 2020-10-09T17:29:26.099 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17", 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.572591972Z", 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.100 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.101 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.616842_114.1", 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.102 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.616842253Z", 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.103 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.104 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.620605_115.1", 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.620605579Z", 2020-10-09T17:29:26.105 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.622618_116.1", 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18", 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.622618991Z", 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.108 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.109 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.110 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.626418_117.1", 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18", 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.626418341Z", 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.111 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.645966_118.1", 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.113 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.645966499Z", 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.651155_119.1", 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.116 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.651155609Z", 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.117 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.655670_120.1", 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.655670493Z", 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.119 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.121 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.661706_121.1", 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.661706240Z", 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.123 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.673791_122.1", 2020-10-09T17:29:26.124 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.673791703Z", 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.681384_123.1", 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.127 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.681384931Z", 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.128 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.974811_154.1", 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.974811077Z", 2020-10-09T17:29:26.130 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.131 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.132 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.978899_165.1", 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.978899378Z", 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.133 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.135 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.980746_166.1", 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19", 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.980746411Z", 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.136 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.137 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.984814_167.1", 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.138 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19", 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.984814622Z", 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.139 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.990937_168.1", 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.990937161Z", 2020-10-09T17:29:26.141 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.142 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.996213_169.1", 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.996213796Z", 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.144 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.146 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.295175_170.1", 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.295175489Z", 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.148 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.299816_191.1", 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.149 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.299816303Z", 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.301833_192.1", 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20", 2020-10-09T17:29:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.301833065Z", 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.305660_193.1", 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20", 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.305660950Z", 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264301.829073_194.1", 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.829073153Z", 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264301.840172_201.1", 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.160 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.840172924Z", 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264356.587328_202.1", 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.163 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.587328671Z", 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.164 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264356.599102_209.1", 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.599102815Z", 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.166 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:26.167 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.657350_220.1", 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.657350424Z", 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.169 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.170 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.662094_223.1", 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:26.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.662094551Z", 2020-10-09T17:29:26.173 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.173 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.173 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.173 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.664114_224.1", 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.175 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21", 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.664114737Z", 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.668177_225.1", 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21", 2020-10-09T17:29:26.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.668177389Z", 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.179 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.694791_226.1", 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.694791618Z", 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.181 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.183 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.705509_227.1", 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.705509786Z", 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.185 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.707607_228.1", 2020-10-09T17:29:26.186 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22", 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.707607520Z", 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.711488_229.1", 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22", 2020-10-09T17:29:26.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.711488882Z", 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.190 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.765125_230.1", 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.765125952Z", 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.192 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.769697_231.1", 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.769697460Z", 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.195 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.196 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.771452_232.1", 2020-10-09T17:29:26.197 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23", 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.771452268Z", 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.775560_233.1", 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23", 2020-10-09T17:29:26.200 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.775560119Z", 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.797874_234.1", 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.797874711Z", 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.203 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.205 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.805041_235.1", 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.805041511Z", 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.809433_236.1", 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.208 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.809433778Z", 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.209 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.813654_237.1", 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.813654996Z", 2020-10-09T17:29:26.211 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.212 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.825509_238.1", 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.825509079Z", 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.214 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.216 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.832028_239.1", 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.832028783Z", 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.217 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.216367_240.1", 2020-10-09T17:29:26.219 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.216367795Z", 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.221041_255.1", 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.221041216Z", 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.223 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.222904_256.1", 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24", 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.222904032Z", 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.225 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.227 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.227259_257.1", 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24", 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.227259403Z", 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.228 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.229 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.252949_258.1", 2020-10-09T17:29:26.230 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.252949731Z", 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.257345_259.1", 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.257345653Z", 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.234 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.259289_260.1", 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25", 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.259289030Z", 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.236 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.262905_261.1", 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25", 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.262905010Z", 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.239 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.307988_262.1", 2020-10-09T17:29:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.307988531Z", 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.312877_263.1", 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.312877842Z", 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.315071_264.1", 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26", 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.315071641Z", 2020-10-09T17:29:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.319616_265.1", 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26", 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.319616392Z", 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.251 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.341664_266.1", 2020-10-09T17:29:26.252 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.341664786Z", 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.253 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.254 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.257 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.347803_267.1", 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.347803294Z", 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.258 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.259 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.351673_268.1", 2020-10-09T17:29:26.260 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.351673675Z", 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.360647_269.1", 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:26.263 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.360647470Z", 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.264 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.370574_270.1", 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.370574457Z", 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.266 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.267 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.375594_271.1", 2020-10-09T17:29:26.268 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.375594201Z", 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.391493_272.1", 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.271 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.391493979Z", 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.272 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.399765_273.1", 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.399765168Z", 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.274 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.415879_274.1", 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.276 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.415879375Z", 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.277 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.420739_275.1", 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.420739980Z", 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.279 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.281 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.427787_276.1", 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.427787757Z", 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.432960_277.1", 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:26.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.432960647Z", 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:26.285 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.204746_283.1", 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.204746328Z", 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.287 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.209341_296.1", 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.289 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.209341959Z", 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.290 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.211337_297.1", 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27", 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.211337050Z", 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.292 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.293 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.215528_298.1", 2020-10-09T17:29:26.294 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27", 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.215528132Z", 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.242661_299.1", 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:26.297 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.242661766Z", 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.298 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.248232_300.1", 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.248232520Z", 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.300 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.301 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.250316_301.1", 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.302 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28", 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.250316394Z", 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.303 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.304 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.254254_302.1", 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28", 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.254254294Z", 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.305 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.306 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.307 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.298996_303.1", 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.298996399Z", 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.308 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.303839_304.1", 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.310 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.303839237Z", 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.311 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.305882_305.1", 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29", 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.305882506Z", 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.313 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.314 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.309980_306.1", 2020-10-09T17:29:26.315 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29", 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.309980653Z", 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.316 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.338582_307.1", 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.338582404Z", 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.318 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.319 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.320 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.344508_308.1", 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.344508600Z", 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.321 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.322 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.393075_309.1", 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.393075237Z", 2020-10-09T17:29:26.323 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.324 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.399422_310.1", 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.399422139Z", 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.326 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.327 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.458912_311.1", 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.328 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.458912136Z", 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.329 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.467949_312.1", 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.467949620Z", 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.331 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.547125_313.1", 2020-10-09T17:29:26.333 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.547125552Z", 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.334 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.552027_314.1", 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.552027100Z", 2020-10-09T17:29:26.336 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.337 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.338 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.554041_315.1", 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30", 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.554041904Z", 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.558805_316.1", 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30", 2020-10-09T17:29:26.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.558805398Z", 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:26.342 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.584159_317.1", 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.584159383Z", 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.344 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.345 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.346 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.588880_318.1", 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.588880019Z", 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.347 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.348 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.590910_319.1", 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31", 2020-10-09T17:29:26.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.590910718Z", 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.350 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.594932_320.1", 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31", 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.594932708Z", 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.352 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.353 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.649758_321.1", 2020-10-09T17:29:26.354 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.649758180Z", 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.355 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.654484_322.1", 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.654484032Z", 2020-10-09T17:29:26.357 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.359 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.656473_323.1", 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32", 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.656473591Z", 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.360 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.361 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.660761_324.1", 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32", 2020-10-09T17:29:26.362 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.660761391Z", 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.363 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.686330_325.1", 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.686330593Z", 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.365 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.367 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.692341_326.1", 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.692341773Z", 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.368 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.744293_327.1", 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:26.370 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.744293216Z", 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.371 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.755410_328.1", 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.755410652Z", 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.373 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.374 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.827943_329.1", 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.375 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.827943489Z", 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.376 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.832753_330.1", 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.832753298Z", 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.378 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.379 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.909951_331.1", 2020-10-09T17:29:26.380 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.909951412Z", 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.381 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.914593_332.1", 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.914593130Z", 2020-10-09T17:29:26.383 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.385 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.916589_333.1", 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33", 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.916589104Z", 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.386 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.387 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.920637_334.1", 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33", 2020-10-09T17:29:26.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.920637835Z", 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:26.389 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.950486_335.1", 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.950486877Z", 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.391 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.392 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.956282_336.1", 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.393 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.956282207Z", 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.394 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.958961_337.1", 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34", 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.958961739Z", 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.396 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.967075_338.1", 2020-10-09T17:29:26.398 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34", 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.967075872Z", 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.018870_339.1", 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.018870930Z", 2020-10-09T17:29:26.401 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.402 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.403 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.022759_340.1", 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.022759559Z", 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.404 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.405 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.024635_341.1", 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35", 2020-10-09T17:29:26.406 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.024635168Z", 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.407 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.028418_342.1", 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35", 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.028418911Z", 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.409 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.410 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.048923_343.1", 2020-10-09T17:29:26.411 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.048923618Z", 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.412 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.055329_344.1", 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.055329261Z", 2020-10-09T17:29:26.414 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.416 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264424.106351_345.1", 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:04.106351971Z", 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264424.117605_346.1", 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:04.117605225Z", 2020-10-09T17:29:26.419 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:26.420 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.177021_347.1", 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.177021241Z", 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.422 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.189008_348.1", 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:26.424 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.189008779Z", 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.425 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.426 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.266549_349.1", 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.266549024Z", 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.427 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.271211_350.1", 2020-10-09T17:29:26.429 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.271211187Z", 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.273170_351.1", 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36", 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.273170349Z", 2020-10-09T17:29:26.432 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:26.433 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.277147_352.1", 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36", 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.277147722Z", 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.435 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.436 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.306602_353.1", 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:26.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.306602896Z", 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.438 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.311387_354.1", 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.311387019Z", 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.440 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.441 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.313355_355.1", 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.442 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37", 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.313355999Z", 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.443 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.317179_356.1", 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37", 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.317179014Z", 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.445 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.365415_357.1", 2020-10-09T17:29:26.447 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.365415525Z", 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.371789_358.1", 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.371789680Z", 2020-10-09T17:29:26.450 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:26.451 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.452 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.374833_359.1", 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38", 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.374833790Z", 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.453 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.454 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.387176_360.1", 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.455 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38", 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.387176842Z", 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.457 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.949266_361.1", 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.458 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.949266551Z", 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.459 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.953535_364.1", 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.953535193Z", 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.461 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.955454_365.1", 2020-10-09T17:29:26.463 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39", 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.955454424Z", 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.465 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.959356_366.1", 2020-10-09T17:29:26.466 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39", 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.959356816Z", 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.983754_367.1", 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.983754988Z", 2020-10-09T17:29:26.469 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:26.470 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.471 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.988786_368.1", 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.988786029Z", 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.472 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.473 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.991106_369.1", 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40", 2020-10-09T17:29:26.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.991106852Z", 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.475 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.995242_370.1", 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40", 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.995242709Z", 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.477 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.041964_371.1", 2020-10-09T17:29:26.479 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.041964955Z", 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.048895_372.1", 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.048895642Z", 2020-10-09T17:29:26.482 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:26.483 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.484 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.051484_373.1", 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41", 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.051484910Z", 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.485 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.486 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.056998_374.1", 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41", 2020-10-09T17:29:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.056998203Z", 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:26.489 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:26.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:26.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:26.492 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:26.531 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:26.753 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:26.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.753 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263785.865052_62.1", 2020-10-09T17:29:26.753 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "user", 2020-10-09T17:29:26.754 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:26.754 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.865052749Z", 2020-10-09T17:29:26.754 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.754 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.756 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.756 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.756 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.757 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.757 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263785.872885_64.1", 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "user", 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885001Z", 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.758 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.759 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.733887_49.1", 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.760 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.733887684Z", 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.762 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.738653_53.1", 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.763 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.738653976Z", 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.765 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.766 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.766 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.740167_54.1", 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1", 2020-10-09T17:29:26.767 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.740167740Z", 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.768 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.743542_55.1", 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-1", 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.743542118Z", 2020-10-09T17:29:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.772 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.764831_56.1", 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.764831074Z", 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.773 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.774 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.774 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.774 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:26.774 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.775 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.775 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.769319_57.1", 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.776 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.769319759Z", 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.777 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.778 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:26.778 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.778 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.771065_58.1", 2020-10-09T17:29:26.779 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2", 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.771065035Z", 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.780 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.781 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.782 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.782 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263908.774595_59.1", 2020-10-09T17:29:26.782 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-2", 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:28.774595327Z", 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.783 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.784 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.123803_60.1", 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.123803579Z", 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.786 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.788 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.127933_61.1", 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.127933618Z", 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.790 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.791 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.791 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.129769_62.1", 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3", 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.129769137Z", 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.793 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.794 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.794 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263912.133520_63.1", 2020-10-09T17:29:26.794 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-3", 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:32.133520501Z", 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.795 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.796 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.356884_64.1", 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:26.797 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.356884810Z", 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.798 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.361606_66.1", 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.800 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.361606375Z", 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.801 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.802 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.363464_67.1", 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4", 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.363464549Z", 2020-10-09T17:29:26.803 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.804 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.805 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.367510_68.1", 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-4", 2020-10-09T17:29:26.806 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.367510128Z", 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:26.807 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.386611_69.1", 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.386611433Z", 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.809 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.810 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.391360_70.1", 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.811 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.391360027Z", 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.812 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.813 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.814 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.814 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.393032_71.1", 2020-10-09T17:29:26.814 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5", 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.393032564Z", 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.815 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.816 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.397175_72.1", 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-5", 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.397175593Z", 2020-10-09T17:29:26.817 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:26.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.820 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.440554_73.1", 2020-10-09T17:29:26.820 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.820 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:26.820 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.440554181Z", 2020-10-09T17:29:26.820 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.822 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.444647_74.1", 2020-10-09T17:29:26.823 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.823 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:26.823 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.444647257Z", 2020-10-09T17:29:26.823 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:26.824 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.825 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.446535_75.1", 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6", 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.446535350Z", 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.826 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.827 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263913.450323_76.1", 2020-10-09T17:29:26.828 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-6", 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:33.450323041Z", 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.829 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.830 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.289813_77.1", 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.289813313Z", 2020-10-09T17:29:26.831 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.832 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.832 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.832 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.832 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.833 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.294257_80.1", 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.294257706Z", 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.835 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.836 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.837 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.837 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.837 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.296137_81.1", 2020-10-09T17:29:26.837 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.296137549Z", 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.838 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.839 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.300098_82.1", 2020-10-09T17:29:26.840 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.841 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:26.841 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.300098832Z", 2020-10-09T17:29:26.841 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.841 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.842 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.842 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.842 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.842 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.843 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.843 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.843 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.843 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.318125_83.1", 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.318125513Z", 2020-10-09T17:29:26.844 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:26.845 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.322838_84.1", 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.322838970Z", 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.847 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:26.853 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.854 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.324748_85.1", 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.324748391Z", 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.855 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.856 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.328548_86.1", 2020-10-09T17:29:26.857 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.328548439Z", 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.858 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.859 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.860 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.374771_87.1", 2020-10-09T17:29:26.861 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.861 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:26.861 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.374771908Z", 2020-10-09T17:29:26.861 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.861 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:26.862 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.863 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.379192_88.1", 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.379192835Z", 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.864 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.865 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.380951_89.1", 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.866 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.380951565Z", 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263916.384719_90.1", 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:36.384719553Z", 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.869 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.871 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.871 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.871 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.673745_91.1", 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.673745199Z", 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.872 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.873 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.874 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.678989_93.1", 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-7", 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.678989138Z", 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.875 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.876 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.878 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.737348_94.1", 2020-10-09T17:29:26.879 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.737348546Z", 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:26.881 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.744698_95.1", 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-8", 2020-10-09T17:29:26.882 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.744698724Z", 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.884 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.844022_96.1", 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.844022609Z", 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.885 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.886 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "remove" 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263917.851306_97.1", 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.887 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-9", 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306494Z", 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.888 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.889 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.192132_44.1", 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.890 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.192132424Z", 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.891 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.196027_61.1", 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.196027568Z", 2020-10-09T17:29:26.893 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.895 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.197885_62.1", 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10", 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.197885358Z", 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.896 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.897 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.898 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.201690_63.1", 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-10", 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.201690295Z", 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.899 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:26.900 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.901 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.224925_64.1", 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.224925356Z", 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.902 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.904 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.229526_65.1", 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.229526856Z", 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.905 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.906 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.231376_66.1", 2020-10-09T17:29:26.907 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11", 2020-10-09T17:29:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.231376014Z", 2020-10-09T17:29:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.908 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.909 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.235568_67.1", 2020-10-09T17:29:26.910 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-11", 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.235568177Z", 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.911 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.912 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.288647_68.1", 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.288647392Z", 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.913 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.915 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.292922_69.1", 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.292922614Z", 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.917 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.918 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.295857_70.1", 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12", 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.295857593Z", 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.919 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.920 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263919.301783_71.1", 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-12", 2020-10-09T17:29:26.921 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:39.301783945Z", 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:26.922 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.923 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.394203_80.1", 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.394203930Z", 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.924 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.925 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.398851_85.1", 2020-10-09T17:29:26.926 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.398851813Z", 2020-10-09T17:29:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.927 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.928 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.401101_86.1", 2020-10-09T17:29:26.929 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13", 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.401101583Z", 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.405020_87.1", 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-13", 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.405020770Z", 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.932 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.426024_88.1", 2020-10-09T17:29:26.934 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.426024492Z", 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.936 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.937 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.431035_89.1", 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.431035284Z", 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.938 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.939 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.433179_90.1", 2020-10-09T17:29:26.940 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14", 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.433179868Z", 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.942 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.437532_91.1", 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-14", 2020-10-09T17:29:26.943 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.437532851Z", 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.945 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.946 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.485384_92.1", 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.485384682Z", 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.947 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.948 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.489772_93.1", 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.489772073Z", 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.950 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.491502_94.1", 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15", 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.491502823Z", 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.952 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:26.953 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263930.495381_95.1", 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-15", 2020-10-09T17:29:26.955 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:50.495381334Z", 2020-10-09T17:29:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.956 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.529296_96.1", 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.958 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.529296713Z", 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.959 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.960 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.533107_107.1", 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.533107387Z", 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.961 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.962 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.534931_108.1", 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16", 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.534931376Z", 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.964 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:26.965 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.538686_109.1", 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16", 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.538686724Z", 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.967 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.968 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.560642_110.1", 2020-10-09T17:29:26.969 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.560642819Z", 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.970 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.565909_111.1", 2020-10-09T17:29:26.972 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.565909467Z", 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.974 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.975 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.568371_112.1", 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17", 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.568371015Z", 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.976 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.977 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.572591_113.1", 2020-10-09T17:29:26.978 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17", 2020-10-09T17:29:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.572591972Z", 2020-10-09T17:29:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.979 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:26.980 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.981 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.616842_114.1", 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.616842253Z", 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.982 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.983 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.984 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.620605_115.1", 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:26.985 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.620605579Z", 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.986 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.622618_116.1", 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.988 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18", 2020-10-09T17:29:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.622618991Z", 2020-10-09T17:29:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.989 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.990 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.626418_117.1", 2020-10-09T17:29:26.991 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18", 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.626418341Z", 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:26.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.993 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.993 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:26.993 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.994 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.645966_118.1", 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.645966499Z", 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.995 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:26.997 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:26.997 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:26.997 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:26.998 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.651155_119.1", 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.651155609Z", 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:26.999 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.001 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.655670_120.1", 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.655670493Z", 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.002 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.003 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.003 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.003 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.004 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.661706_121.1", 2020-10-09T17:29:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.661706240Z", 2020-10-09T17:29:27.005 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.006 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.673791_122.1", 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.673791703Z", 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.009 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602263979.681384_123.1", 2020-10-09T17:29:27.010 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:19:39.681384931Z", 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.012 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.974811_154.1", 2020-10-09T17:29:27.013 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.014 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.014 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.974811077Z", 2020-10-09T17:29:27.014 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.014 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.014 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.016 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.978899_165.1", 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.978899378Z", 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.017 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.018 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.980746_166.1", 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.019 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19", 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.980746411Z", 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.020 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.021 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.022 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.984814_167.1", 2020-10-09T17:29:27.022 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19", 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.984814622Z", 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.023 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.024 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.990937_168.1", 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.025 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.990937161Z", 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.026 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.027 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264162.996213_169.1", 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:42.996213796Z", 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.028 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.030 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.295175_170.1", 2020-10-09T17:29:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.295175489Z", 2020-10-09T17:29:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.299816_191.1", 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.299816303Z", 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.034 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.301833_192.1", 2020-10-09T17:29:27.036 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20", 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.301833065Z", 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.038 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.039 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.039 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.039 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.039 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264292.305660_193.1", 2020-10-09T17:29:27.039 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20", 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:24:52.305660950Z", 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.041 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.041 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:27.041 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.042 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264301.829073_194.1", 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:01.829073153Z", 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.043 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.044 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264301.840172_201.1", 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.045 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:01.840172924Z", 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.047 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.048 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264356.587328_202.1", 2020-10-09T17:29:27.048 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:56.587328671Z", 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.050 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.050 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.050 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.051 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264356.599102_209.1", 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:25:56.599102815Z", 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.052 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.053 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.657350_220.1", 2020-10-09T17:29:27.054 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.657350424Z", 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.055 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.662094_223.1", 2020-10-09T17:29:27.057 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.662094551Z", 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.059 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.059 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.059 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.059 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.060 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.664114_224.1", 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21", 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.664114737Z", 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.062 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.668177_225.1", 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21", 2020-10-09T17:29:27.063 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.668177389Z", 2020-10-09T17:29:27.064 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.064 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.064 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.064 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.065 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.694791_226.1", 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.066 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.694791618Z", 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.068 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.069 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.705509_227.1", 2020-10-09T17:29:27.069 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.069 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.705509786Z", 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.070 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.707607_228.1", 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22", 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.707607520Z", 2020-10-09T17:29:27.072 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.073 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.073 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.073 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.073 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.074 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.711488_229.1", 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22", 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.711488882Z", 2020-10-09T17:29:27.075 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.076 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.077 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.765125_230.1", 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.765125952Z", 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.078 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.080 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.769697_231.1", 2020-10-09T17:29:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.769697460Z", 2020-10-09T17:29:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.082 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.083 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.771452_232.1", 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23", 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.771452268Z", 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.084 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.086 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.087 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.775560_233.1", 2020-10-09T17:29:27.087 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.087 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23", 2020-10-09T17:29:27.087 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.775560119Z", 2020-10-09T17:29:27.088 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.088 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.089 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.089 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.089 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:27.090 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.090 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.091 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.091 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.091 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.092 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.092 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.797874_234.1", 2020-10-09T17:29:27.092 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.092 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.093 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.797874711Z", 2020-10-09T17:29:27.093 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.093 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.093 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.094 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.094 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.095 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.095 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.095 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.096 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.096 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.096 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.097 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.097 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.805041_235.1", 2020-10-09T17:29:27.097 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.098 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.098 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.805041511Z", 2020-10-09T17:29:27.098 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.098 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.099 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.099 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.099 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.100 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.101 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.101 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.101 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.101 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.102 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.809433_236.1", 2020-10-09T17:29:27.102 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.103 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.103 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.809433778Z", 2020-10-09T17:29:27.103 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.104 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.104 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.104 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.106 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.106 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.106 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.107 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.107 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.107 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.813654_237.1", 2020-10-09T17:29:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.813654996Z", 2020-10-09T17:29:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.109 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.109 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.109 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.110 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.110 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.110 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.111 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.111 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.112 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.112 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.112 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.825509_238.1", 2020-10-09T17:29:27.113 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.113 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.113 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.825509079Z", 2020-10-09T17:29:27.114 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.114 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.114 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.114 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.115 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.115 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.116 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.116 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.116 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.117 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.118 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264401.832028_239.1", 2020-10-09T17:29:27.118 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.118 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.118 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:41.832028783Z", 2020-10-09T17:29:27.119 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.119 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.119 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.120 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.120 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.120 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.121 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.121 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.121 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.123 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.123 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.216367_240.1", 2020-10-09T17:29:27.123 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.123 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.124 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.216367795Z", 2020-10-09T17:29:27.124 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.124 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.124 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.125 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.125 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.126 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.126 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.127 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.127 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.128 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.128 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.221041_255.1", 2020-10-09T17:29:27.128 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.128 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.129 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.221041216Z", 2020-10-09T17:29:27.129 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.129 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.130 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.130 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.130 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.131 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.131 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.131 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.132 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.133 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.133 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.222904_256.1", 2020-10-09T17:29:27.133 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.134 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24", 2020-10-09T17:29:27.134 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.222904032Z", 2020-10-09T17:29:27.134 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.134 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.135 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.135 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.136 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.136 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:27.136 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.137 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.137 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.137 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.137 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.138 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.138 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.138 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.227259_257.1", 2020-10-09T17:29:27.139 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.139 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24", 2020-10-09T17:29:27.139 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.227259403Z", 2020-10-09T17:29:27.139 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.140 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.140 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.141 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:27.141 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.142 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.142 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.142 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.143 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.252949_258.1", 2020-10-09T17:29:27.144 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.144 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.144 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.252949731Z", 2020-10-09T17:29:27.144 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.145 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.145 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.146 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.146 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.146 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.147 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.147 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.147 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.148 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.148 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.149 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.257345_259.1", 2020-10-09T17:29:27.149 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.149 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.149 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.257345653Z", 2020-10-09T17:29:27.150 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.150 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.150 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.150 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.151 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.151 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.152 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.152 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.152 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.154 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.259289_260.1", 2020-10-09T17:29:27.154 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.154 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25", 2020-10-09T17:29:27.155 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.259289030Z", 2020-10-09T17:29:27.155 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.155 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.155 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.156 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.156 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.156 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:27.157 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.157 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.157 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.158 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.158 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.159 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.159 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.262905_261.1", 2020-10-09T17:29:27.159 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.159 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25", 2020-10-09T17:29:27.160 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.262905010Z", 2020-10-09T17:29:27.160 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.160 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.160 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.161 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.161 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:27.162 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.162 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.163 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.163 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.163 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.164 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.164 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.307988_262.1", 2020-10-09T17:29:27.164 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.164 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.165 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.307988531Z", 2020-10-09T17:29:27.165 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.165 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.166 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.166 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.166 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.167 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.167 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.168 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.168 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.168 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.169 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.169 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.312877_263.1", 2020-10-09T17:29:27.169 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.170 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.170 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.312877842Z", 2020-10-09T17:29:27.170 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.170 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.171 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.171 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.172 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.315071_264.1", 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26", 2020-10-09T17:29:27.173 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.315071641Z", 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.174 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.175 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.319616_265.1", 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26", 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.319616392Z", 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.177 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.341664_266.1", 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.178 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.341664786Z", 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.179 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.347803_267.1", 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.347803294Z", 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.181 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.182 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.351673_268.1", 2020-10-09T17:29:27.183 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.351673675Z", 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.184 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.360647_269.1", 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.360647470Z", 2020-10-09T17:29:27.186 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.187 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.188 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.370574_270.1", 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.370574457Z", 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.190 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.375594_271.1", 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.375594201Z", 2020-10-09T17:29:27.191 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.193 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.391493_272.1", 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.391493979Z", 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.194 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.195 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.399765_273.1", 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.399765168Z", 2020-10-09T17:29:27.196 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:27.197 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.198 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.415879_274.1", 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.415879375Z", 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.199 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.200 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.420739_275.1", 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.420739980Z", 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.202 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.203 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.427787_276.1", 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.427787757Z", 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.204 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.205 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.206 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264412.432960_277.1", 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:26:52.432960647Z", 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 3 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.208 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.204746_283.1", 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.204746328Z", 2020-10-09T17:29:27.209 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.210 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.211 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.209341_296.1", 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.209341959Z", 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.212 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.213 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.211337_297.1", 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.214 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27", 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.211337050Z", 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.215 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.215528_298.1", 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27", 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.215528132Z", 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.217 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.218 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.242661_299.1", 2020-10-09T17:29:27.219 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.242661766Z", 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.221 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.248232_300.1", 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:27.222 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.248232520Z", 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.223 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.224 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.250316_301.1", 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28", 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.250316394Z", 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.225 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.226 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.254254_302.1", 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.227 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28", 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.254254294Z", 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.298996_303.1", 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.298996399Z", 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.230 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.231 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.303839_304.1", 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.232 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.303839237Z", 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.233 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.305882_305.1", 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29", 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.305882506Z", 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.235 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.236 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.309980_306.1", 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.237 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29", 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.309980653Z", 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.238 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.239 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.338582_307.1", 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.338582404Z", 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.240 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.241 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.344508_308.1", 2020-10-09T17:29:27.242 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.344508600Z", 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.243 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.244 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.393075_309.1", 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.393075237Z", 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.245 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.399422_310.1", 2020-10-09T17:29:27.247 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.399422139Z", 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.249 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.458912_311.1", 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.458912136Z", 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.250 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.467949_312.1", 2020-10-09T17:29:27.252 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.467949620Z", 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.547125_313.1", 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.547125552Z", 2020-10-09T17:29:27.255 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.257 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.552027_314.1", 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.552027100Z", 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.258 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.554041_315.1", 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30", 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.554041904Z", 2020-10-09T17:29:27.260 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.262 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.558805_316.1", 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30", 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.558805398Z", 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.263 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.264 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.584159_317.1", 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:27.265 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.584159383Z", 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.266 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.588880_318.1", 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.588880019Z", 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.268 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.269 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.590910_319.1", 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31", 2020-10-09T17:29:27.270 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.590910718Z", 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.271 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.272 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.594932_320.1", 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31", 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.594932708Z", 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.273 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.274 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.649758_321.1", 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.275 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.649758180Z", 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.654484_322.1", 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.654484032Z", 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.278 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.279 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.656473_323.1", 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32", 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.656473591Z", 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.660761_324.1", 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32", 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.660761391Z", 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.283 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.284 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.686330_325.1", 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.285 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.686330593Z", 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.286 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.287 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.692341_326.1", 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.692341773Z", 2020-10-09T17:29:27.288 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.290 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.744293_327.1", 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.744293216Z", 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.291 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.292 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.755410_328.1", 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:27.293 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.755410652Z", 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.827943_329.1", 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.827943489Z", 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.296 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.297 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.832753_330.1", 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.298 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.832753298Z", 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.299 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.909951_331.1", 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.909951412Z", 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.301 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.914593_332.1", 2020-10-09T17:29:27.303 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.914593130Z", 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.305 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.916589_333.1", 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33", 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.916589104Z", 2020-10-09T17:29:27.306 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.308 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.920637_334.1", 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33", 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.920637835Z", 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.950486_335.1", 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.950486877Z", 2020-10-09T17:29:27.311 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.313 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.956282_336.1", 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.956282207Z", 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.958961_337.1", 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34", 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.958961739Z", 2020-10-09T17:29:27.316 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.318 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264422.967075_338.1", 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34", 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:02.967075872Z", 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.319 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.018870_339.1", 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:27.321 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.018870930Z", 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.322 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.022759_340.1", 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.022759559Z", 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.324 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.024635_341.1", 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35", 2020-10-09T17:29:27.326 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.024635168Z", 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.327 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.328 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.028418_342.1", 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35", 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.028418911Z", 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.329 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.330 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.048923_343.1", 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:27.331 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.048923618Z", 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.332 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.333 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264423.055329_344.1", 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:03.055329261Z", 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.334 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264424.106351_345.1", 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.336 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:04.106351971Z", 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.337 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.338 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264424.117605_346.1", 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:04.117605225Z", 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.339 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.341 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.342 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.177021_347.1", 2020-10-09T17:29:27.342 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.342 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:27.342 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.177021241Z", 2020-10-09T17:29:27.342 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.343 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.344 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.189008_348.1", 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.189008779Z", 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.345 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 2 2020-10-09T17:29:27.346 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.347 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.266549_349.1", 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.266549024Z", 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.349 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.271211_350.1", 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.271211187Z", 2020-10-09T17:29:27.350 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:27.351 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.352 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.273170_351.1", 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36", 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.273170349Z", 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.354 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.277147_352.1", 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-36", 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.277147722Z", 2020-10-09T17:29:27.355 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:27.356 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.357 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.306602_353.1", 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.306602896Z", 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.358 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.359 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.311387_354.1", 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:27.360 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.311387019Z", 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:27.361 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.313355_355.1", 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37", 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.313355999Z", 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.363 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.364 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.317179_356.1", 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.365 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-37", 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.317179014Z", 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.366 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.365415_357.1", 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.365415525Z", 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.368 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.369 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.371789_358.1", 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.370 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.371789680Z", 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.371 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.374833_359.1", 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38", 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.374833790Z", 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.373 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.375 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264425.387176_360.1", 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-38", 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:05.387176842Z", 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.949266_361.1", 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.949266551Z", 2020-10-09T17:29:27.378 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:27.379 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.380 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.953535_364.1", 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.953535193Z", 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.381 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.382 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.955454_365.1", 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39", 2020-10-09T17:29:27.383 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.955454424Z", 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.384 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.959356_366.1", 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-39", 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.959356816Z", 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.387 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.983754_367.1", 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.388 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.983754988Z", 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.389 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.988786_368.1", 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.988786029Z", 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.391 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.392 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.991106_369.1", 2020-10-09T17:29:27.393 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40", 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.991106852Z", 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.394 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264426.995242_370.1", 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-40", 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:06.995242709Z", 2020-10-09T17:29:27.396 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.398 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.041964_371.1", 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.041964955Z", 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.400 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.048895_372.1", 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.048895642Z", 2020-10-09T17:29:27.401 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.403 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.051484_373.1", 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41", 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.051484910Z", 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.404 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.405 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264427.056998_374.1", 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:27.406 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-41", 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.056998203Z", 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:27.407 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.408 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:27.409 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:27.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:27.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:27.411 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:27.452 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:27.452 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:27.674 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:27.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.675 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263785.865052_62.1", 2020-10-09T17:29:27.675 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-10-09T17:29:27.675 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:27.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.865052749Z", 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.676 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263785.872885_64.1", 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "user", 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "rgw-multisite-test-user", 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885001Z", 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.678 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Fw6htcLGoLrsEEoj5uhQ2Em", 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.679 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.680 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.733887_49.1", 2020-10-09T17:29:27.681 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.681 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:27.681 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.733887684Z", 2020-10-09T17:29:27.681 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.681 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.682 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.683 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.738653_53.1", 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1", 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.738653976Z", 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.684 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M_a_QkztkopMyDVzuYWse5A", 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.740167_54.1", 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.686 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1", 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.740167740Z", 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.687 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.743542_55.1", 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-1", 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.743542118Z", 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.689 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.690 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.690 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.690 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.690 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_4DSARVCi9Hc4Pm2eci-RwBW", 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.764831_56.1", 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.764831074Z", 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.692 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.694 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.769319_57.1", 2020-10-09T17:29:27.695 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.695 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2", 2020-10-09T17:29:27.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.769319759Z", 2020-10-09T17:29:27.695 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.695 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_rbRmMdlgEUnWHP0oOtQqcGZ", 2020-10-09T17:29:27.696 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.697 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.771065_58.1", 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2", 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.771065035Z", 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.698 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.699 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263908.774595_59.1", 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-2", 2020-10-09T17:29:27.700 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:28.774595327Z", 2020-10-09T17:29:27.701 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.701 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.701 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.701 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BXJ-3nQDPx4XodFlYl1gU5o", 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.123803_60.1", 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.123803579Z", 2020-10-09T17:29:27.703 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:27.704 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.127933_61.1", 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3", 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.127933618Z", 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.706 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MxvAGsPJce4PCHomyL8eO6O", 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.707 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.129769_62.1", 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.708 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3", 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.129769137Z", 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.709 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263912.133520_63.1", 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-3", 2020-10-09T17:29:27.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:32.133520501Z", 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.712 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hyBO3cT37KkavFh59Yo8YuP", 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.356884_64.1", 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.356884810Z", 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.714 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.715 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.361606_66.1", 2020-10-09T17:29:27.716 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4", 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.361606375Z", 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_yyMfzPk6-a8kI-s9UO6xChi", 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.363464_67.1", 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4", 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.363464549Z", 2020-10-09T17:29:27.719 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:27.720 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.367510_68.1", 2020-10-09T17:29:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-4", 2020-10-09T17:29:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.367510128Z", 2020-10-09T17:29:27.722 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_EzjsEik0yTXLxPAwWA8iyiZ", 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.724 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.386611_69.1", 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.386611433Z", 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.725 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.726 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.391360_70.1", 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.727 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5", 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.391360027Z", 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.728 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_t5URKEqJmuu_IVME8ruSS-t", 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.393032_71.1", 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5", 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.393032564Z", 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.730 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:27.731 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.397175_72.1", 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-5", 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.397175593Z", 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.733 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ToWXZE6g3zZhwjptjA3_3_A", 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.734 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.440554_73.1", 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.735 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.440554181Z", 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.736 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.444647_74.1", 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6", 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.444647257Z", 2020-10-09T17:29:27.738 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_LfzmvOpaInZUUcrjd92tXge", 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.446535_75.1", 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6", 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.446535350Z", 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.741 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.743 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263913.450323_76.1", 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-6", 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:33.450323041Z", 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.744 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_916vk-8DdMjKurYMvXIuJ91", 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.289813_77.1", 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.746 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.289813313Z", 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.747 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.294257_80.1", 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7", 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.294257706Z", 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.749 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Kvfr96ZL1DzakhdEwLQ6mht", 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.751 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.296137_81.1", 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.296137549Z", 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.752 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.753 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.300098_82.1", 2020-10-09T17:29:27.754 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.300098832Z", 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.318125_83.1", 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.757 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.318125513Z", 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.758 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.322838_84.1", 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8", 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.322838970Z", 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.760 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Lv0y5c9NaClscUr9O9EGRvC", 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.762 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.762 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.324748_85.1", 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.324748391Z", 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.763 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.328548_86.1", 2020-10-09T17:29:27.765 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.328548439Z", 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.767 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.374771_87.1", 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.374771908Z", 2020-10-09T17:29:27.768 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:27.769 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.379192_88.1", 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9", 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.379192835Z", 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.771 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_W1R2vol_ysCsTczq3R-i5PD", 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.772 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.773 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.380951_89.1", 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.380951565Z", 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.774 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.775 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263916.384719_90.1", 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:27.776 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:36.384719553Z", 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.777 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.673745_91.1", 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.673745199Z", 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.779 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.678989_93.1", 2020-10-09T17:29:27.781 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-7", 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.678989138Z", 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.782 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ow2qYxmS7BV7vPG8oh4Gief", 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.783 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.785 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.737348_94.1", 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.786 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.737348546Z", 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.787 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.744698_95.1", 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-8", 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.744698724Z", 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.789 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jngRjhBAhd1C9-XrZ3ZGrVt", 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.844022_96.1", 2020-10-09T17:29:27.791 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.844022609Z", 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.793 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "remove" 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263917.851306_97.1", 2020-10-09T17:29:27.794 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-9", 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306494Z", 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.795 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_h9gkAGgUmHCRQVY5u6gWGWR", 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.192132_44.1", 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:27.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.192132424Z", 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.798 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.196027_61.1", 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10", 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.196027568Z", 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.800 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jT5UEhshWsXfPqh2VciywAC", 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.801 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.197885_62.1", 2020-10-09T17:29:27.802 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10", 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.197885358Z", 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.803 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.804 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.201690_63.1", 2020-10-09T17:29:27.805 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-10", 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.201690295Z", 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_fnXQ3Z5H3JcWN1CxitzEhn2", 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.224925_64.1", 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.224925356Z", 2020-10-09T17:29:27.808 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:27.809 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.229526_65.1", 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11", 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.229526856Z", 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.811 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_b8kYjg8iuAwsWamEImqsGY2", 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.812 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.231376_66.1", 2020-10-09T17:29:27.813 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11", 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.231376014Z", 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.235568_67.1", 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-11", 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.235568177Z", 2020-10-09T17:29:27.816 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_uAFSr6hUlNfczsmJZB5A0mu", 2020-10-09T17:29:27.817 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.818 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.288647_68.1", 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.288647392Z", 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.819 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.820 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.292922_69.1", 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12", 2020-10-09T17:29:27.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.292922614Z", 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.822 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_FbOoHRp19akZDQ89FodIRXv", 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.295857_70.1", 2020-10-09T17:29:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12", 2020-10-09T17:29:27.824 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.295857593Z", 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:27.825 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263919.301783_71.1", 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-12", 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:39.301783945Z", 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.827 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_RA-qdzEjDTETclftcnNIQrB", 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.394203_80.1", 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.829 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.394203930Z", 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.830 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.398851_85.1", 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13", 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.398851813Z", 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.832 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_R22pB25VQbvEB8A_OGUG9q0", 2020-10-09T17:29:27.833 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.401101_86.1", 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13", 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.401101583Z", 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.835 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.836 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.405020_87.1", 2020-10-09T17:29:27.837 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-13", 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.405020770Z", 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_7zcmtZmO3xaQBOODDZZES0i", 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.839 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.426024_88.1", 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:27.840 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.426024492Z", 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.841 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.431035_89.1", 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14", 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.431035284Z", 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.843 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.844 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.844 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.844 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.844 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_-FCFm_77Y2J3TvEVrkEvIKj", 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.845 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.433179_90.1", 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14", 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.433179868Z", 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.846 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.847 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.437532_91.1", 2020-10-09T17:29:27.848 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-14", 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.437532851Z", 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5HchZNrjiWXqW8X3adNxHlj", 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.485384_92.1", 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:27.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.485384682Z", 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:27.852 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.489772_93.1", 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15", 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.489772073Z", 2020-10-09T17:29:27.854 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c6k8Snh1jDMdaJ6RmTngs0g", 2020-10-09T17:29:27.855 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.856 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.491502_94.1", 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15", 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.491502823Z", 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.857 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263930.495381_95.1", 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.859 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-15", 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:50.495381334Z", 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.860 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cL4pnKuQFLAgAPAdQ922_JV", 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.529296_96.1", 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.529296713Z", 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.533107_107.1", 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.533107387Z", 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.534931_108.1", 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16", 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.534931376Z", 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.538686_109.1", 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16", 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.538686724Z", 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YkYKr7C5I_Dvx-pzr6juFxw", 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.560642_110.1", 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.560642819Z", 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.874 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.874 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.874 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.874 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.565909_111.1", 2020-10-09T17:29:27.875 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.565909467Z", 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.877 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.883 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.568371_112.1", 2020-10-09T17:29:27.884 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.884 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17", 2020-10-09T17:29:27.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.568371015Z", 2020-10-09T17:29:27.884 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.884 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.886 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.572591_113.1", 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17", 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.572591972Z", 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_mrvqrcBAjvRSZDr11XN4bTo", 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.616842_114.1", 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.616842253Z", 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.890 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.891 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.892 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.892 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.620605_115.1", 2020-10-09T17:29:27.892 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.892 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.892 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.620605579Z", 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.893 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.622618_116.1", 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.895 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18", 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.622618991Z", 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.896 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.626418_117.1", 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18", 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.626418341Z", 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.898 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5CbmthUYIaziXycMw-hxNgG", 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.900 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.645966_118.1", 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.645966499Z", 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.651155_119.1", 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16", 2020-10-09T17:29:27.903 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.651155609Z", 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.904 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_C1N1JhxZ8qDxEQUqRz-0cJj", 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.905 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.655670_120.1", 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.655670493Z", 2020-10-09T17:29:27.906 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.907 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.661706_121.1", 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17", 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.661706240Z", 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.909 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AjPcib6THm8EE_z_CyA4oK6", 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.910 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.673791_122.1", 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.911 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.673791703Z", 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.912 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602263979.681384_123.1", 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18", 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:19:39.681384931Z", 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.914 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.915 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.915 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.915 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.915 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_wjYBzI5lZMYPB6WlpeSpB3O", 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.974811_154.1", 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.974811077Z", 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.917 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.978899_165.1", 2020-10-09T17:29:27.919 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.978899378Z", 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.920 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.921 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.980746_166.1", 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19", 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.980746411Z", 2020-10-09T17:29:27.922 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.924 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.924 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.984814_167.1", 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19", 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.984814622Z", 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.925 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_WNJIeWtETwsLj0NJMihUX2R", 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.926 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.990937_168.1", 2020-10-09T17:29:27.927 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.990937161Z", 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.928 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264162.996213_169.1", 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19", 2020-10-09T17:29:27.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:42.996213796Z", 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.931 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HoyU01y5EnPiXE-HgVANMg0", 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.295175_170.1", 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.295175489Z", 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.933 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.935 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.299816_191.1", 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.299816303Z", 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.936 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.301833_192.1", 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.938 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20", 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.301833065Z", 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.939 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264292.305660_193.1", 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20", 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:24:52.305660950Z", 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.941 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_S7Zsn0pCv1DQn7TsyD5DyS7", 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.942 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264301.829073_194.1", 2020-10-09T17:29:27.943 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.829073153Z", 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.944 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.945 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264301.840172_201.1", 2020-10-09T17:29:27.946 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:01.840172924Z", 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.947 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264356.587328_202.1", 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.587328671Z", 2020-10-09T17:29:27.949 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.950 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.951 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264356.599102_209.1", 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20", 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:25:56.599102815Z", 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.952 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_6nxvhkPWJGJH0-BmdnEAOnV", 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.657350_220.1", 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.954 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.657350424Z", 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.955 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.662094_223.1", 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.662094551Z", 2020-10-09T17:29:27.957 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.958 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.959 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.664114_224.1", 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21", 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.664114737Z", 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.960 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.961 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.962 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.668177_225.1", 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21", 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.668177389Z", 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.963 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qvx368-HGTJLuDuD9UgCouz", 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.965 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.694791_226.1", 2020-10-09T17:29:27.965 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.694791618Z", 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.705509_227.1", 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.968 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.705509786Z", 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.969 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.707607_228.1", 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22", 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.707607520Z", 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.971 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.711488_229.1", 2020-10-09T17:29:27.973 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.974 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22", 2020-10-09T17:29:27.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.711488882Z", 2020-10-09T17:29:27.974 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.974 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_UEQUuwxxZhw7kZpduT8YNPj", 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.976 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.765125_230.1", 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.765125952Z", 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.977 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.978 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.769697_231.1", 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.769697460Z", 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.980 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.981 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.771452_232.1", 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23", 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.771452268Z", 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.982 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.983 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.775560_233.1", 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:27.984 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23", 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.775560119Z", 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.985 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_cJPpHXj-mnb6BEoYcbwkeUJ", 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.986 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.797874_234.1", 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.797874711Z", 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.987 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.805041_235.1", 2020-10-09T17:29:27.989 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21", 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.805041511Z", 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DQ6qK_4vHhcfNoi9xnpfnRH", 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.809433_236.1", 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.809433778Z", 2020-10-09T17:29:27.992 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.993 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.813654_237.1", 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22", 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.813654996Z", 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.995 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_pC6JJ4dHsDILyxUbZXGGEiK", 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.996 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.825509_238.1", 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:27.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.825509079Z", 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:27.998 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:27.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264401.832028_239.1", 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23", 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:41.832028783Z", 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.000 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_r4arv7-Sztv-YHuciO5XtZz", 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.216367_240.1", 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.002 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.216367795Z", 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.003 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.221041_255.1", 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.221041216Z", 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.005 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.222904_256.1", 2020-10-09T17:29:28.007 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24", 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.222904032Z", 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.008 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.227259_257.1", 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24", 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.227259403Z", 2020-10-09T17:29:28.010 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Z5-Sqny7nY79d02cTk9P3ol", 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.012 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.252949_258.1", 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.252949731Z", 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.257345_259.1", 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.257345653Z", 2020-10-09T17:29:28.015 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.016 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.259289_260.1", 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25", 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.259289030Z", 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.018 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.019 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.262905_261.1", 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.020 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25", 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.262905010Z", 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.021 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_dJP69-ruUKlbpjy3Ovvcu5c", 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.307988_262.1", 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.307988531Z", 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.023 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.312877_263.1", 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.025 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.312877842Z", 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.026 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.315071_264.1", 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26", 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.315071641Z", 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.028 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.319616_265.1", 2020-10-09T17:29:28.030 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26", 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.319616392Z", 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZSwA94l1hTDwYvV9XV4hXnd", 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.341664_266.1", 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.341664786Z", 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.347803_267.1", 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.347803294Z", 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.351673_268.1", 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.351673675Z", 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.360647_269.1", 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24", 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.360647470Z", 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_KaQzSsGQkbJXPQhOCj7CYZ5", 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.370574_270.1", 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.370574457Z", 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.375594_271.1", 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.375594201Z", 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.048 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.391493_272.1", 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.391493979Z", 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.049 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.399765_273.1", 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25", 2020-10-09T17:29:28.051 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.399765168Z", 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.052 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qzn21f9HZLlgvozOBrRll00", 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.415879_274.1", 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.415879375Z", 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.054 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.420739_275.1", 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.056 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.420739980Z", 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.057 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.427787_276.1", 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.427787757Z", 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.059 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.061 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264412.432960_277.1", 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26", 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:26:52.432960647Z", 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_c3sCs2WjQ5-XT0AhEsH2W7L", 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 3 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.204746_283.1", 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.204746328Z", 2020-10-09T17:29:28.064 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.065 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.209341_296.1", 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.209341959Z", 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.067 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.211337_297.1", 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27", 2020-10-09T17:29:28.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.211337050Z", 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.070 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.215528_298.1", 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27", 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.215528132Z", 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.072 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_kmEPBWzdq35tE9QQ4j9rFO2", 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.073 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.242661_299.1", 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.074 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.242661766Z", 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.075 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.248232_300.1", 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.248232520Z", 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.077 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.079 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.250316_301.1", 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28", 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.250316394Z", 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.080 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.081 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.254254_302.1", 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28", 2020-10-09T17:29:28.082 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.254254294Z", 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.083 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_5MKQyecCSEY52zqPxufPeUe", 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.298996_303.1", 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.298996399Z", 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.085 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.303839_304.1", 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.087 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.303839237Z", 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.088 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.305882_305.1", 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29", 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.305882506Z", 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.090 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.309980_306.1", 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.092 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29", 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.309980653Z", 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.093 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_gZW7YPG9EICPay96JxE6SRb", 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.338582_307.1", 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.338582404Z", 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.095 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.344508_308.1", 2020-10-09T17:29:28.097 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-27:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27", 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.344508600Z", 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_3z8sULQVNGE-ASzNUGWiVJv", 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.393075_309.1", 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.393075237Z", 2020-10-09T17:29:28.100 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.101 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.102 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.399422_310.1", 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-28:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28", 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.399422139Z", 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.103 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_iQHK8bBKPoBgKzb9RGiPbl9", 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.104 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.458912_311.1", 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:28.105 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.458912136Z", 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.106 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.107 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.467949_312.1", 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-29:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29", 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.467949620Z", 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.108 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_HQiKIbn34O-zAWdwXkrIBXm", 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.109 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.547125_313.1", 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.110 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.547125552Z", 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.111 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.112 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.552027_314.1", 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.552027100Z", 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.113 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.114 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.554041_315.1", 2020-10-09T17:29:28.115 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30", 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.554041904Z", 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.116 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.558805_316.1", 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30", 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.558805398Z", 2020-10-09T17:29:28.118 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1GiqzehjVXaQaTe_vSwHbL2", 2020-10-09T17:29:28.119 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.120 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.584159_317.1", 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.584159383Z", 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.121 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.588880_318.1", 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:28.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.588880019Z", 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.124 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.125 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.590910_319.1", 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31", 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.590910718Z", 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.126 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.127 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.594932_320.1", 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.128 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31", 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.594932708Z", 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.129 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_8WvscyawSvhkwmz3dOoG9V-", 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.649758_321.1", 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.649758180Z", 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.131 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.654484_322.1", 2020-10-09T17:29:28.133 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.654484032Z", 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.134 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.656473_323.1", 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32", 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.656473591Z", 2020-10-09T17:29:28.136 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.138 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.660761_324.1", 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32", 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.660761391Z", 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_lL2gTRkaUYm0w127SMq3cHO", 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.686330_325.1", 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.141 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.686330593Z", 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.142 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.692341_326.1", 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-30:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30", 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.692341773Z", 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.144 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ru2fOY6nmv91f38FAH6gKTZ", 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.146 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.744293_327.1", 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.744293216Z", 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.147 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.148 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.755410_328.1", 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-31:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31", 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.755410652Z", 2020-10-09T17:29:28.149 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_MIbfz--gGbCMXs_17r1Mslg", 2020-10-09T17:29:28.150 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.827943_329.1", 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.827943489Z", 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.152 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.153 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.832753_330.1", 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.154 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-32:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32", 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.832753298Z", 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.155 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A7YnrvyA1t8S4XhXYCKp772", 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.909951_331.1", 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.909951412Z", 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.157 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.914593_332.1", 2020-10-09T17:29:28.159 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.914593130Z", 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.916589_333.1", 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33", 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.916589104Z", 2020-10-09T17:29:28.162 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:28.163 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.164 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.920637_334.1", 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33", 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.920637835Z", 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.165 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_GrhlAI7y6Kzd1BZEBD9UrxR", 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.950486_335.1", 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:28.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.950486877Z", 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.168 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.956282_336.1", 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.956282207Z", 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.170 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.171 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.958961_337.1", 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.172 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34", 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.958961739Z", 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.173 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264422.967075_338.1", 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34", 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:02.967075872Z", 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.175 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qSMFPAoNYK5Kehhvut81lph", 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.176 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.018870_339.1", 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.177 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.018870930Z", 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.178 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.022759_340.1", 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.022759559Z", 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.180 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.024635_341.1", 2020-10-09T17:29:28.182 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35", 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.024635168Z", 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.183 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.028418_342.1", 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35", 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.028418911Z", 2020-10-09T17:29:28.185 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Ga_yci22Tk_TKSxSYdrxX8D", 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.187 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.048923_343.1", 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.048923618Z", 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264423.055329_344.1", 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-33:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33", 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:03.055329261Z", 2020-10-09T17:29:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_m68h7ol3cPmxU4iUUE9C7QN", 2020-10-09T17:29:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264424.106351_345.1", 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:04.106351971Z", 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264424.117605_346.1", 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-34:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34", 2020-10-09T17:29:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:04.117605225Z", 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_hQ0dTrgbfHv45LsFGVjaIB3", 2020-10-09T17:29:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.177021_347.1", 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.177021241Z", 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.199 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.189008_348.1", 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.200 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-35:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35", 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.189008779Z", 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.201 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_ZdWLJhTjuBqBFp_7paI1nib", 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 2 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.202 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.266549_349.1", 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.266549024Z", 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.203 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.271211_350.1", 2020-10-09T17:29:28.205 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36", 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.271211187Z", 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.206 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_xScmFJAq6jEycu7NURWn5S6", 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.273170_351.1", 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36", 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.273170349Z", 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.208 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.210 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.277147_352.1", 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-36", 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.277147722Z", 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_DsUEafjAtU-3yI3L6DVRz6O", 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.306602_353.1", 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.306602896Z", 2020-10-09T17:29:28.213 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:28.214 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.215 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.311387_354.1", 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37", 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.311387019Z", 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.216 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_bciMyluaDZoNgQUw1dlAF7u", 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.313355_355.1", 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37", 2020-10-09T17:29:28.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.313355999Z", 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:28.219 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.317179_356.1", 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-37", 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.317179014Z", 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.221 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_zi0PSRVUOKAFq97_Zki74An", 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.365415_357.1", 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.223 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.365415525Z", 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.224 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.371789_358.1", 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38", 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.371789680Z", 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.226 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Q1V9J0iwjB0cs7-h4dh8NDD", 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.374833_359.1", 2020-10-09T17:29:28.228 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38", 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.374833790Z", 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.229 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264425.387176_360.1", 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-38", 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:05.387176842Z", 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.231 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_TrKplz8S9yip1nhlPFJccj7", 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.949266_361.1", 2020-10-09T17:29:28.233 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.949266551Z", 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.953535_364.1", 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39", 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.953535193Z", 2020-10-09T17:29:28.236 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_A2pdjQQGraz3s25c5ZUY74h", 2020-10-09T17:29:28.237 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.955454_365.1", 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39", 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.955454424Z", 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.239 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.959356_366.1", 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-39", 2020-10-09T17:29:28.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.959356816Z", 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.242 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_s-AlRYhXf8VfSSSLJwMDkWi", 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.983754_367.1", 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.983754988Z", 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.244 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.245 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.988786_368.1", 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.246 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40", 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.988786029Z", 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.247 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_k3FPzNjjo943sGA5nvg83Ik", 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.991106_369.1", 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40", 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.991106852Z", 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.249 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.250 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264426.995242_370.1", 2020-10-09T17:29:28.251 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-40", 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:06.995242709Z", 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.252 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_BPkcILzoz7Bf5GBUXrE-uH-", 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.041964_371.1", 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.041964955Z", 2020-10-09T17:29:28.254 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.256 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.048895_372.1", 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41", 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.048895642Z", 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.257 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_M4RIqWsbdzbO3ifMcEgLTEC", 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.051484_373.1", 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41", 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.051484910Z", 2020-10-09T17:29:28.259 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:28.260 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.261 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264427.056998_374.1", 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-41", 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.056998203Z", 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.262 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_1cQCircMB3d_ueQp6mWsOe0", 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.264 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:28.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.268 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:28.309 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:28.540 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.903049_334.1", 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.903049069Z", 2020-10-09T17:29:28.541 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:28.542 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.543 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.907466_335.1", 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.907466753Z", 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.544 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.545 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.910067_336.1", 2020-10-09T17:29:28.546 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:28.547 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42", 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.910067916Z", 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.548 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.549 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.917091_337.1", 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42", 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.917091951Z", 2020-10-09T17:29:28.550 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:28.551 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.552 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.969167_338.1", 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.969167595Z", 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.553 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.554 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.554 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.554 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:28.554 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.976828_339.1", 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.976828198Z", 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.556 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.557 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.557 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.557 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.558 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.979191_340.1", 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43", 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.979191296Z", 2020-10-09T17:29:28.559 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:28.560 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:28.562 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.984679_341.1", 2020-10-09T17:29:28.562 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:28.562 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43", 2020-10-09T17:29:28.562 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.984679785Z", 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.563 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:28.564 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:28.564 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:28.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:28.564 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:28.571 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:28.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.572 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:28.572 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:28.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.575 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:28.612 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:28.612 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:28.830 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:28.830 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.903049_334.1", 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.903049069Z", 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.831 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.832 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.907466_335.1", 2020-10-09T17:29:28.833 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.907466753Z", 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.834 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.835 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.910067_336.1", 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.836 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42", 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.910067916Z", 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.837 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.917091_337.1", 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42", 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.917091951Z", 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.839 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.841 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.841 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.969167_338.1", 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.969167595Z", 2020-10-09T17:29:28.842 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.843 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.844 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.976828_339.1", 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.976828198Z", 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.845 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.846 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.979191_340.1", 2020-10-09T17:29:28.847 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43", 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.979191296Z", 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.984679_341.1", 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43", 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.984679785Z", 2020-10-09T17:29:28.850 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:28.851 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:28.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:28.852 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:28.859 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:28.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:28.859 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:28.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:28.862 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:28.906 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:28.907 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:29.126 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:29.126 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.126 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.903049_334.1", 2020-10-09T17:29:29.126 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.903049069Z", 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.127 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.128 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.907466_335.1", 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.129 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.907466753Z", 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.131 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.910067_336.1", 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42", 2020-10-09T17:29:29.132 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.910067916Z", 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.134 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.917091_337.1", 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-42", 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.917091951Z", 2020-10-09T17:29:29.135 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:29.136 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.969167_338.1", 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.969167595Z", 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.138 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.140 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.976828_339.1", 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.976828198Z", 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.141 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.142 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.979191_340.1", 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.143 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43", 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.979191296Z", 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264494.984679_341.1", 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-43", 2020-10-09T17:29:29.146 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:28:14.984679785Z", 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.147 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.148 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:29.148 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.148 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.157 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.158 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.158 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.158 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.158 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:29.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.160 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:29.199 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:29.200 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:29.405 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:29.405 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.405 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.903049_334.1", 2020-10-09T17:29:29.405 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.903049069Z", 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:29.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.907466_335.1", 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1", 2020-10-09T17:29:29.408 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.907466753Z", 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.409 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0dNaTJKW6-53Cm_9HfHI0Yx", 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.910067_336.1", 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42", 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.910067916Z", 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.411 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.412 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.917091_337.1", 2020-10-09T17:29:29.413 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-42", 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.917091951Z", 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_AxhAJCBF3Vwn8oSltKCEsq_", 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.969167_338.1", 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.969167595Z", 2020-10-09T17:29:29.416 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.418 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.976828_339.1", 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43:bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2", 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.976828198Z", 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.419 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_qqO21rNqS3S_5ANRWP6pUU0", 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.420 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.979191_340.1", 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43", 2020-10-09T17:29:29.421 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.979191296Z", 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.422 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264494.984679_341.1", 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-43", 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:28:14.984679785Z", 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:29.424 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_jxa2BLZeNT_q96yx1UHJ2wB", 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:29.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:29.426 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:29.434 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:29.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:29.435 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:29.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.438 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:29.478 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:29.706 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:29.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.706 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.173307_149.1", 2020-10-09T17:29:29.707 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.707 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:29.707 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.173307040Z", 2020-10-09T17:29:29.707 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.707 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.708 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.708 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.708 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.708 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:29.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.709 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.709 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.710 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.711 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.177887_150.1", 2020-10-09T17:29:29.711 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.711 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:29.712 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.177887632Z", 2020-10-09T17:29:29.712 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.712 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.713 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.713 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.713 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.714 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:29.714 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.715 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.715 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.715 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.716 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.716 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.179702_151.1", 2020-10-09T17:29:29.716 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44", 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.179702764Z", 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.718 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.183648_152.1", 2020-10-09T17:29:29.719 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44", 2020-10-09T17:29:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.183648364Z", 2020-10-09T17:29:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.720 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.221299_153.1", 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.221299553Z", 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.723 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:29.724 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.725 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.225747_154.1", 2020-10-09T17:29:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:29.726 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.225747139Z", 2020-10-09T17:29:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.727 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.728 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.227862_155.1", 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45", 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.227862064Z", 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.730 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:29.731 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:29.732 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.231925_156.1", 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45", 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.231925806Z", 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:29.733 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:29.734 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.737 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:29.737 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:29.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:29.739 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:29.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:29.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:30.002 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.173307_149.1", 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.173307040Z", 2020-10-09T17:29:30.003 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.004 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.005 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.177887_150.1", 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.177887632Z", 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.006 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.179702_151.1", 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.008 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.179702764Z", 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.009 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.183648_152.1", 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.183648364Z", 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.011 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.221299_153.1", 2020-10-09T17:29:30.013 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.221299553Z", 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.225747_154.1", 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.225747139Z", 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.017 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.227862_155.1", 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.227862064Z", 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.019 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.020 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.231925_156.1", 2020-10-09T17:29:30.021 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.231925806Z", 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.023 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.023 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.023 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.023 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.031 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:30.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:30.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:30.034 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:30.304 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:30.304 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.173307_149.1", 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.173307040Z", 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.305 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.307 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.177887_150.1", 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.177887632Z", 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.308 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:30.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.179702_151.1", 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.310 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.179702764Z", 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.311 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.312 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.183648_152.1", 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.183648364Z", 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.314 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.221299_153.1", 2020-10-09T17:29:30.315 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.221299553Z", 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.316 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.225747_154.1", 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.225747139Z", 2020-10-09T17:29:30.318 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.319 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.320 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.227862_155.1", 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.227862064Z", 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.321 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.322 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "write" 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "1_1602264556.231925_156.1", 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: "section": "bucket", 2020-10-09T17:29:30.323 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:16.231925806Z", 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "data": { 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "read_version": { 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "", 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 0 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.324 INFO:teuthology.orchestra.run.smithi060.stdout: "write_version": { 2020-10-09T17:29:30.325 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.325 INFO:teuthology.orchestra.run.smithi060.stdout: "ver": 1 2020-10-09T17:29:30.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:30.325 INFO:teuthology.orchestra.run.smithi060.stdout: "status": { 2020-10-09T17:29:30.335 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "complete" 2020-10-09T17:29:30.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:30.336 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:30.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:30.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:30.339 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:30.380 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:30.380 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:30.604 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.173307_149.1", 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.173307040Z", 2020-10-09T17:29:30.605 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.606 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.177887_150.1", 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1", 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.177887632Z", 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.608 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_Xl_UF_8fvm4mXX55VVrrUBe", 2020-10-09T17:29:30.609 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.179702_151.1", 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.179702764Z", 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.611 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.612 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.183648_152.1", 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-44", 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.183648364Z", 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.614 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_sM6wybjS1lPDjQ45x0uT3ph", 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.615 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.221299_153.1", 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.616 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.221299553Z", 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.617 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.225747_154.1", 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket.instance", 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2", 2020-10-09T17:29:30.619 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.225747139Z", 2020-10-09T17:29:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.620 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_YhkPYPawtN9UWSh9UW4W--L", 2020-10-09T17:29:30.621 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.622 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.227862_155.1", 2020-10-09T17:29:30.623 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.624 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.227862064Z", 2020-10-09T17:29:30.624 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.624 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.625 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.625 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.625 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.625 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.626 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.626 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "write" 2020-10-09T17:29:30.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:30.627 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "1_1602264556.231925_156.1", 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "section": "bucket", 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "wypfiu-45", 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:16.231925806Z", 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "data": { 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "read_version": { 2020-10-09T17:29:30.628 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "", 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 0 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: "write_version": { 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: "tag": "_0nS298rMYuTCIN2n8S-kNjh", 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: "ver": 1 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:30.629 INFO:teuthology.orchestra.run.smithi184.stdout: "status": { 2020-10-09T17:29:30.632 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "complete" 2020-10-09T17:29:30.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:30.633 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:30.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:30.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:30.636 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:30.674 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:30.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:30.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:30.957 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:30.993 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:30.994 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:31.399 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:31.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:31.400 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:31.435 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:31.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:31.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:31.709 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:31.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:31.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:32.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.104 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:32.142 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:32.398 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:32.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.401 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:32.440 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:32.440 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:32.701 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:32.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.704 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:32.742 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:32.743 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:32.981 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:32.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'eb32ec64-e1b3-4311-8266-3310720f134e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:32.984 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period eb32ec64-e1b3-4311-8266-3310720f134e --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:33.023 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:33.023 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:33.257 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:33.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.260 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:33.300 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:33.300 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:33.562 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:33.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.565 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:33.603 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:33.603 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:33.857 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:33.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:33.859 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:33.898 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:34.156 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:34.158 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f137cd21-eb16-49e0-9fe5-a2877c0fdad4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.159 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f137cd21-eb16-49e0-9fe5-a2877c0fdad4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:34.195 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:34.195 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:34.462 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:34.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.466 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:34.504 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:34.504 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:34.767 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:34.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:34.770 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:34.810 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:34.811 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:35.060 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:35.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:35.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:35.063 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:35.102 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:35.367 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-10-09T17:29:35.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:35.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'e380f3c0-dc07-4443-880b-21f94e64f4aa', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:35.370 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period e380f3c0-dc07-4443-880b-21f94e64f4aa --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:35.408 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:35.408 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:35.671 INFO:teuthology.orchestra.run.smithi184.stdout:[] 2020-10-09T17:29:35.673 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-10-09T17:29:35.674 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-46 2020-10-09T17:29:35.709 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-47 2020-10-09T17:29:35.815 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-48 2020-10-09T17:29:36.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.016 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:36.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:36.257 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:29:36.282 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:36.282 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:36.283 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.284 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:29:36.285 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:35.843730Z" 2020-10-09T17:29:36.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.285 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:36.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:29:36.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.287 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:36.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:36.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:36.581 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:36.581 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:36.581 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:36.581 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:36.582 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:36.582 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:36.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:36.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:36.582 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:36.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:36.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:36.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:29:36.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:36.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:36.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:36.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:36.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:36.588 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:36.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:36.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:36.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:36.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:29:36.591 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:29:36.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:36.591 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:36.592 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:29:36.592 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:29:36.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.594 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:36.638 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:36.638 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:36.837 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:36.837 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:36.837 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:36.837 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:36.837 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.838 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:36.839 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:36.840 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:36.841 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:36.842 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.843 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:36.844 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:36.845 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:29:36.846 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:29:36.846 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:36.846 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:36.847 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:29:36.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:29:36.848 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:36.849 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:36.890 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:36.891 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:37.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:37.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:37.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.153 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:37.153 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.153 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:37.153 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:29:37.153 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.156 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:29:37.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.158 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:37.158 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.160 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:37.161 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:29:37.161 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:29:37.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.161 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:37.162 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:29:37.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.164 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:37.205 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:37.234 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:37.468 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:37.468 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:37.469 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:37.470 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:37.471 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:29:37.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.473 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:37.515 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:37.516 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:37.779 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:37.779 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:37.779 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:37.782 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:37.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.786 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:37.787 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:37.788 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:29:37.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:37.790 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:37.830 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:38.098 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:38.099 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.099 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:38.099 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:38.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.100 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.100 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:38.101 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:38.101 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.101 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.101 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:38.102 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:38.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.102 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.103 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:38.103 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:38.103 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.103 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:38.105 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:29:38.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.107 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:38.199 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:38.201 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:38.453 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:38.454 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:38.454 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:38.454 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:38.455 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:38.455 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 2964130218513816294 2020-10-09T17:29:38.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.455 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:38.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:38.456 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:38.457 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:38.457 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:38.457 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.458 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.458 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:38.458 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:38.458 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:38.458 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:38.459 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:38.459 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:38.459 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:38.459 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:38.459 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:38.460 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:38.461 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:38.461 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:38.461 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:38.461 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:38.461 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.462 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.462 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:38.462 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:38.463 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:38.464 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:38.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.464 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.464 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:38.464 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.465 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:38.465 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:29:38.465 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:29:38.465 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.465 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:38.467 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:29:38.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.469 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:38.558 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:38.559 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:38.807 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:38.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.807 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:38.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:38.810 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:38.810 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:38.811 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T17:29:38.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:38.812 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:38.856 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:38.856 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:39.121 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:39.122 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.123 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:39.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.126 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:39.128 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:39.129 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:29:39.130 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:29:39.130 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.130 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:39.131 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T17:29:39.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.132 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:39.219 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:39.220 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:39.482 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:39.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:39.482 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:27:07.137946Z" 2020-10-09T17:29:39.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.483 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:29:36.618320Z" 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:39.484 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:39.485 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-09T17:29:39.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.487 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:39.533 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:39.782 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:39.783 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:39.783 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:39.784 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:39.784 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:39.784 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 13602710843643633186 2020-10-09T17:29:39.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:39.785 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:39.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:39.785 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:39.785 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:39.785 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.786 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:39.786 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:39.786 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:39.786 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:39.786 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.137946Z" 2020-10-09T17:29:39.787 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:39.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:39.787 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:39.788 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:39.788 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.788 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:29:39.789 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:39.789 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:39.789 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:39.789 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:29:39.790 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:39.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:39.790 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:39.791 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:39.791 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.792 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:39.792 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:39.793 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:39.793 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:39.794 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:36.618320Z" 2020-10-09T17:29:39.794 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:39.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:39.795 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:39.795 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:39.796 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:39.796 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:29:39.797 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:39.797 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:39.797 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:39.798 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:29:39.798 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.798 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.799 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:39.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:39.799 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:39.800 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:29:39.800 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:29:39.800 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:39.800 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:39.803 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-10-09T17:29:39.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:39.805 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:39.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:39.846 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:40.096 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:27:07.137946Z" 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:29:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:29:36.618320Z" 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:40.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:29:40.099 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:29:40.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:40.099 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:40.100 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-09T17:29:40.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.102 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:40.145 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:40.408 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:40.408 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 743555352260775809 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:40.409 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:40.410 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.137946Z" 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:40.411 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:40.413 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:36.618320Z" 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:40.414 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.415 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.416 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:40.417 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-10-09T17:29:40.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.419 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:40.499 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:40.738 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.048693Z" 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:40.739 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.254661Z" 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:40.740 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:29:40.741 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:29:40.741 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:40.741 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:40.742 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-09T17:29:40.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:40.743 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:40.827 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:40.827 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:41.081 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:29:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:29:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:29:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:29:41.081 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 6335292525355606852 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:41.082 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:27:07.048693Z" 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:41.084 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:41.085 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:29:36.254661Z" 2020-10-09T17:29:41.086 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:29:41.087 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.088 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:29:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:29:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:29:41.089 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.089 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:29:41.090 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-10-09T17:29:41.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.091 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:41.174 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:41.175 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:41.428 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:41.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.429 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:41.429 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.048693Z" 2020-10-09T17:29:41.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.429 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:41.430 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.254661Z" 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:41.431 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:41.432 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-09T17:29:41.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.433 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:41.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:41.478 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:41.752 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 601039138999268403 2020-10-09T17:29:41.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.754 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.048693Z" 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:41.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:41.756 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.848255Z" 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:41.757 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.254661Z" 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:41.759 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:41.759 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:29:41.759 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:41.762 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:29:41.763 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005300", 2020-10-09T17:29:41.763 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:29:41.763 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:29:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:29:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:23:16.614483Z" 2020-10-09T17:29:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:41.765 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:29:41.766 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:29:41.766 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:29:41.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:41.766 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:41.768 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-10-09T17:29:41.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:41.770 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:41.808 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:42.068 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:42.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:42.069 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:42.069 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:42.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:42.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:42.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:42.071 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:42.071 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:42.071 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:42.071 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:42.071 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:42.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:42.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:42.073 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:42.149 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:42.149 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:42.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:42.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:42.408 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:42.444 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:42.445 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:42.709 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:27:07.137946Z" 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:42.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002773", 2020-10-09T17:29:42.711 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:10.654856Z" 2020-10-09T17:29:42.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:42.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:42.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003968", 2020-10-09T17:29:42.711 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:29:36.618320Z" 2020-10-09T17:29:42.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:42.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:42.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000005566", 2020-10-09T17:29:42.712 INFO:teuthology.orchestra.run.smithi184.stdout: "last_update": "2020-10-09T17:22:54.670570Z" 2020-10-09T17:29:42.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:42.713 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:29:42.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:42.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:42.715 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:42.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:42.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:43.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:43.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:29:43.076 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:29:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:43.112 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:43.386 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:29:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:29:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:29:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:29:43.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:29:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:29:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000005832", 2020-10-09T17:29:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:23:16.591954Z" 2020-10-09T17:29:43.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:29:43.389 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:29:43.390 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.391 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:43.469 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:43.470 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:43.743 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-10-09T17:29:43.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.745 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:43.787 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:43.787 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:43.808 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:29:43.804+0000 7f77c26fe0c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:29:43.808 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:29:43.804+0000 7f77c26fe0c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:29:43.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:29:43.804+0000 7f77c26fe0c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:29:43.811 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:29:43.804+0000 7f77c26fe0c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:29:43.821 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:43.822 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:29:43.822 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:29:43.822 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:29:43.822 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:29:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:29:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:29:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:29:43.823 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:29:43.824 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:29:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:29:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:29:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:29:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:29:43.825 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:29:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:29:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:43.826 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:29:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:29:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:29:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:29:43.827 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:29:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:29:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:29:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:43.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:29:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:29:43.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:29:43.830 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:29:43.831 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:43.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:29:43.834 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:29:43.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:29:43.915 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:29:44.174 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:29:44.244 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:29:44.244 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 2, 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:29:44.245 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:29:44.246 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:29:44.247 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:29:44.248 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:29:44.249 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:29:44.250 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:29:44.251 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:29:44.253 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:29:44.254 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:29:44.255 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:29:44.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:29:44.257 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:29:44.258 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:29:44.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.259 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:29:44.268 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:29:44.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.268 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:29:44.270 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:29:44.271 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:29:44.272 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:29:44.273 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:29:44.273 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:29:44.273 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:29:44.273 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:29:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:29:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:29:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:29:44.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:29:44.275 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:29:44.275 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:29:44.276 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-10-09T17:30:44.277 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-49 2020-10-09T17:30:44.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:44.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:44.368 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:30:44.410 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:30:44.411 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:30:44.629 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:30:44.662 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:30:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:30:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:30:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:30:44.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:30:44.664 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:30:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:30:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:30:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:30:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:30:44.317066Z" 2020-10-09T17:30:44.665 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:30:44.666 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:30:44.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:30:44.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:44.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:44.669 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:30:44.710 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:30:44.710 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:30:44.972 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:30:44.972 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:30:44.972 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:30:44.972 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:30:44.972 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:30:44.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:44.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:44.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:44.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:44.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:44.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:30:44.980 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:30:44.981 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:30:44.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:44.981 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:30:44.982 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:30:49.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:49.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:49.984 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:30:50.024 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:30:50.024 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:30:50.249 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:30:50.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:30:50.250 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:30:50.250 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:30:50.250 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:30:50.251 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:30:50.251 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:50.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:50.252 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:30:50.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:50.252 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:30:50.252 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:50.253 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:50.253 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:30:50.253 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:50.254 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:30:50.254 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:50.254 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:30:50.254 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:50.255 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:50.257 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:50.258 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:50.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:30:50.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:30:50.261 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:30:50.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:50.261 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:30:50.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:30:55.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:55.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:30:55.265 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:30:55.306 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:30:55.307 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:30:55.574 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:30:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:30:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:30:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:30:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:30:55.575 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:30:55.576 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:55.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:30:55.578 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:30:55.579 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:55.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:55.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:55.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:30:55.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:30:55.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:30:55.582 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:30:55.584 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:30:55.585 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:00.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:00.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:00.588 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:00.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:00.630 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:00.904 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:00.905 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:00.906 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:00.908 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:00.909 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:00.910 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:00.911 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.912 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:00.913 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:00.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:05.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:05.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:05.918 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:05.959 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:05.960 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:06.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:06.230 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:06.231 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:06.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:06.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:06.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:06.235 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:06.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:06.237 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:06.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:06.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:06.239 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:06.239 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:11.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:11.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:11.244 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:11.284 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:11.285 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:11.544 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:11.545 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:11.546 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:11.547 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:11.548 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:11.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:11.550 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:11.551 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:11.552 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:11.553 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:11.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:11.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:11.555 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:11.556 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:16.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:16.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:16.559 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:16.601 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:16.602 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:16.857 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:16.857 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:16.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:16.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:16.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:16.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:16.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:16.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:16.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:16.866 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:16.867 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:21.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:21.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:21.869 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:21.910 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:21.910 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:22.149 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:22.149 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:22.150 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:22.151 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:22.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:22.153 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:22.154 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:22.155 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:22.156 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:22.158 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:22.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:27.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:27.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:27.162 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:27.201 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:27.202 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:27.463 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:27.464 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:27.464 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:27.464 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:27.464 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:27.464 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:27.465 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:27.466 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:27.467 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:27.468 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:27.469 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:27.470 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:27.471 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:27.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:27.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:27.474 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:32.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:32.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:32.476 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:32.518 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:32.519 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:32.781 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:32.781 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:32.781 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:32.781 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:32.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:32.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:32.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:32.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:32.786 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.787 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:32.788 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:32.789 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:32.790 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:32.790 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:32.790 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:32.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:32.791 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:32.792 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:37.794 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:37.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:37.795 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:37.833 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:37.833 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:38.086 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:38.086 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:38.086 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:38.086 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:38.086 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:38.087 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:38.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:38.089 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.090 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:38.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:38.092 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:38.093 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:38.094 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:38.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:38.095 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:38.096 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:43.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:43.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:43.100 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:43.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:43.143 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:43.407 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:43.407 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:43.408 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:43.408 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:43.408 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:43.408 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:43.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:43.410 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:43.410 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:43.410 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:43.412 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:43.412 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:43.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:43.412 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:43.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:43.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:43.414 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:43.414 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:43.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:43.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:43.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:43.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:43.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:43.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:43.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:43.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:43.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:43.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264575.843730_185.1", 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:35.843730Z", 2020-10-09T17:31:43.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:43.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:43.421 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:43.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264644.317066_210.1 target=1_1602264575.843730_185.1 2020-10-09T17:31:44.427 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:31:44.423+0000 7f0c8ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:31:44.431 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:31:44.426+0000 7f1fc17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:31:48.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:48.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:48.427 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:48.468 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:48.468 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:48.722 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:48.723 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:48.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:48.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:48.726 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:48.727 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:48.728 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:48.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.730 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:48.731 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:48.732 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:31:48.733 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:31:48.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:48.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:48.734 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:48.772 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:48.772 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:49.021 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:31:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:31:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:31:49.021 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:31:49.022 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:31:49.023 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:31:49.024 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:31:49.025 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:31:49.026 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:31:49.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:31:49.028 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.029 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:31:49.030 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:31:49.031 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:31:49.032 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:31:49.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:49.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:31:49.033 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:31:49.073 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:49.073 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:49.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:49.340 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:31:49.340 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.341 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:31:49.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.344 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:31:49.344 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:49.349 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:49.349 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:31:49.349 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:31:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:31:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:31:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:31:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.355 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:49.356 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:31:49.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=cc444700-5757-461c-a1a0-0c413b7ba0e8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:31:49.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=cc444700-5757-461c-a1a0-0c413b7ba0e8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:31:49.378 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=cc444700-5757-461c-a1a0-0c413b7ba0e8 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:31:49.438 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:49.438 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:49.464 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:31:49.461+0000 7f3d988320c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:31:49.465 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:31:49.461+0000 7f3d988320c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:31:49.467 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:31:49.461+0000 7f3d988320c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:31:49.468 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:31:49.461+0000 7f3d988320c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:31:49.478 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:31:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:31:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:31:49.478 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:31:49.479 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:31:49.480 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:31:49.481 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.482 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:31:49.483 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:49.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:31:49.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:31:49.485 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:31:49.566 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:31:49.567 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:31:49.838 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:31:49.910 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:31:49.910 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:49.910 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 3, 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:31:49.911 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:31:49.912 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:31:49.913 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:31:49.914 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:31:49.915 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:31:49.916 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:31:49.917 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:31:49.918 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:31:49.919 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8" 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.920 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:31:49.921 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.922 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:49.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:31:49.924 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.935 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:31:49.935 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:31:49.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:31:49.936 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:31:49.937 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:31:49.938 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.939 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:31:49.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:31:49.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:31:49.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:31:49.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:31:49.941 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:31:49.942 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "cc444700-5757-461c-a1a0-0c413b7ba0e8" 2020-10-09T17:32:50.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:32:50.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:32:50.294 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:32:50.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:32:50.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:32:50.362 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:32:50.356+0000 7f8184c040c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:32:50.363 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:32:50.356+0000 7f8184c040c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:32:50.364 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:32:50.360+0000 7f8184c040c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:32:50.365 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:32:50.360+0000 7f8184c040c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:32:50.374 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:32:50.375 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:32:50.375 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:32:50.375 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:32:50.375 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:32:50.376 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:32:50.377 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:32:50.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.380 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:32:50.381 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:32:50.381 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:32:50.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:32:50.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:32:50.383 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:32:50.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:32:50.462 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:32:50.730 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:32:50.806 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:32:50.806 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:32:50.806 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 4, 2020-10-09T17:32:50.806 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:32:50.807 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:32:50.808 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:32:50.809 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:32:50.810 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:32:50.811 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:32:50.812 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:32:50.812 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:32:50.812 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:32:50.813 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:32:50.813 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.813 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.813 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:32:50.814 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:32:50.814 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:32:50.814 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:32:50.814 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:32:50.815 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:32:50.816 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:32:50.816 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:32:50.816 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.816 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:32:50.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.817 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:32:50.818 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:32:50.818 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:32:50.818 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:32:50.818 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.819 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:32:50.819 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:32:50.819 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:32:50.819 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:32:50.819 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8" 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.820 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:32:50.821 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:32:50.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:32:50.823 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:32:50.824 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:32:50.825 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:32:50.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.835 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:32:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:32:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:32:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:32:50.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:32:50.837 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.838 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:32:50.839 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:32:50.840 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:32:50.840 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:32:50.840 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:32:50.840 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:32:50.841 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.841 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:32:50.841 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:32:50.841 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:32:50.841 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:32:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:32:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:32:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:32:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:32:50.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:32:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:32:50.843 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:32:50.843 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:32:50.844 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-10-09T17:33:50.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:33:50.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:33:50.848 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:33:50.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:33:50.887 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:33:50.917 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:33:50.911+0000 7f3cf4f520c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:33:50.917 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:33:50.911+0000 7f3cf4f520c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:33:50.920 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:33:50.915+0000 7f3cf4f520c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:33:50.920 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:33:50.915+0000 7f3cf4f520c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "test-zone2.rgw.meta:root", 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "test-zone2.rgw.control", 2020-10-09T17:33:50.931 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "test-zone2.rgw.log", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2020-10-09T17:33:50.932 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "test-zone2.rgw.otp", 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:33:50.933 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:33:50.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: "compression_type": "random" 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:50.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2020-10-09T17:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:50.936 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:33:50.937 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2020-10-09T17:33:50.937 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:33:50.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:33:50.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:33:50.939 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:33:51.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:33:51.017 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:33:51.277 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:33:51.347 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:33:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:33:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 5, 2020-10-09T17:33:51.348 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:33:51.349 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:33:51.350 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.351 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:33:51.352 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:33:51.353 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.354 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:33:51.355 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:33:51.356 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:33:51.357 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:33:51.358 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.359 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:33:51.360 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:33:51.361 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:33:51.362 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.363 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.364 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:33:51.369 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:33:51.372 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:33:51.373 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:33:51.374 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:33:51.375 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:33:51.376 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-10-09T17:34:51.378 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-10-09T17:34:51.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.382 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:34:51.430 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:51.431 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:34:51.642 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:34:51.689 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:34:51.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:34:51.690 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:34:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:34:51.691 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:34:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:51.692 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:34:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:34:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:51.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:34:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:30:44.317066Z" 2020-10-09T17:34:51.694 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:51.695 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:34:51.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:34:51.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.696 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:34:51.737 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:51.738 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:34:51.980 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:34:51.981 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:51.982 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.983 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:51.984 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:51.985 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:51.986 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:51.987 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.988 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:51.989 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:34:51.990 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:34:51.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:34:51.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:51.992 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:34:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:52.033 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:34:52.287 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:34:52.287 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:34:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:34:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:34:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:34:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:34:52.288 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:34:52.289 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:34:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:34:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:34:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:34:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:34:52.290 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:34:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:34:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:52.291 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:34:52.292 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:34:52.293 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:34:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:34:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:34:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:34:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:34:52.294 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:34:52.295 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:34:52.296 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:34:52.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:34:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:34:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:34:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:34:52.298 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:34:52.298 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:34:52.300 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:34:52.300 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:34:52.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:52.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:34:52.302 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:34:52.343 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:52.391 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:34:52.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:34:52.716 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:34:52.716 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:34:52.716 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:34:52.717 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:34:52.717 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:34:52.717 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:34:52.718 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.718 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:34:52.718 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:52.719 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:34:52.719 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:52.719 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:52.719 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:34:52.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:52.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:34:52.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:52.721 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:34:52.721 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:52.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.722 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.722 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:52.722 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:34:52.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:52.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:52.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:34:52.723 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:52.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:52.724 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:52.724 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:34:52.725 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:52.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:52.726 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:34:52.726 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:52.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:52.727 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:52.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:34:52.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:34:52.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:52.728 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:34:52.729 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:52.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:34:52.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:34:52.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:34:52.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:34:52.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:34:52.731 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:34:52.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:34:52.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.732 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:34:52.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.732 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:34:52.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:34:52.733 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:34:52.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.733 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:34:52.735 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:34:52.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:34:52.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'XB9G80S0Y3NF6E39', '--secret', 'm63w5q18iRriVSoZZk9fSpg1ldWXYCba', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:34:52.737 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key XB9G80S0Y3NF6E39 --secret m63w5q18iRriVSoZZk9fSpg1ldWXYCba --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:34:52.820 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:52.821 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:34:52.888 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:34:52.886+0000 7fcf6e0ec0c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:34:52.888 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:34:52.886+0000 7fcf6e0ec0c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:34:52.889 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:34:52.886+0000 7fcf6e0ec0c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:34:52.890 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:34:52.886+0000 7fcf6e0ec0c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:34:52.899 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:34:52.899 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "d44f7e25-d35f-4de0-8a3b-c9248a88bcf5", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "domain_root": "remove.rgw.meta:root", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "control_pool": "remove.rgw.control", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "gc_pool": "remove.rgw.log:gc", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "lc_pool": "remove.rgw.log:lc", 2020-10-09T17:34:52.900 INFO:teuthology.orchestra.run.smithi184.stdout: "log_pool": "remove.rgw.log", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-10-09T17:34:52.901 INFO:teuthology.orchestra.run.smithi184.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "otp_pool": "remove.rgw.otp", 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "system_key": { 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "access_key": "XB9G80S0Y3NF6E39", 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "secret_key": "m63w5q18iRriVSoZZk9fSpg1ldWXYCba" 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.902 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_pools": [ 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "default-placement", 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "index_pool": "remove.rgw.buckets.index", 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": { 2020-10-09T17:34:52.903 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD": { 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "data_pool": "remove.rgw.buckets.data" 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: "index_type": 0 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:34:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:34:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:34:52.905 INFO:teuthology.orchestra.run.smithi184.stdout: "notif_pool": "remove.rgw.log:notif" 2020-10-09T17:34:52.905 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:34:52.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:34:52.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:34:52.907 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:34:52.985 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:34:52.985 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.062 INFO:teuthology.orchestra.run.smithi184.stderr:Sending period to new master zone cc444700-5757-461c-a1a0-0c413b7ba0e8 2020-10-09T17:35:01.141 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "epoch": 6, 2020-10-09T17:35:01.142 INFO:teuthology.orchestra.run.smithi184.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:35:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": [ 2020-10-09T17:35:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:35:01.143 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:35:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: "", 2020-10-09T17:35:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:35:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.144 INFO:teuthology.orchestra.run.smithi184.stdout: "period_map": { 2020-10-09T17:35:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "zonegroups": [ 2020-10-09T17:35:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.145 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "test-zonegroup", 2020-10-09T17:35:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zonegroup", 2020-10-09T17:35:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: "api_name": "", 2020-10-09T17:35:01.146 INFO:teuthology.orchestra.run.smithi184.stdout: "is_master": "true", 2020-10-09T17:35:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:35:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.147 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames": [], 2020-10-09T17:35:01.148 INFO:teuthology.orchestra.run.smithi184.stdout: "hostnames_s3website": [], 2020-10-09T17:35:01.148 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.148 INFO:teuthology.orchestra.run.smithi184.stdout: "zones": [ 2020-10-09T17:35:01.149 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.149 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:01.149 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone3", 2020-10-09T17:35:01.149 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:35:01.150 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.150 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.150 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:35:01.151 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:35:01.151 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.151 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:35:01.151 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:35:01.152 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.152 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:35:01.152 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:35:01.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.153 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:01.153 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone4", 2020-10-09T17:35:01.154 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:35:01.154 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.154 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.154 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:35:01.155 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:35:01.155 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.155 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:35:01.156 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "pubsub", 2020-10-09T17:35:01.156 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "false", 2020-10-09T17:35:01.156 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [ 2020-10-09T17:35:01.156 INFO:teuthology.orchestra.run.smithi184.stdout: "test-zone1" 2020-10-09T17:35:01.157 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.157 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:35:01.157 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.158 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:01.158 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone2", 2020-10-09T17:35:01.158 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:35:01.159 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.159 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.159 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:35:01.160 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:35:01.160 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.160 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:35:01.160 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:35:01.161 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.161 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:35:01.161 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:35:01.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.162 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "test-zone1", 2020-10-09T17:35:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [ 2020-10-09T17:35:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.163 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:35:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:35:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.164 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:35:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:35:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:35:01.165 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:35:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.166 INFO:teuthology.orchestra.run.smithi184.stdout: "id": "d44f7e25-d35f-4de0-8a3b-c9248a88bcf5", 2020-10-09T17:35:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "remove", 2020-10-09T17:35:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "endpoints": [], 2020-10-09T17:35:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "log_meta": "false", 2020-10-09T17:35:01.167 INFO:teuthology.orchestra.run.smithi184.stdout: "log_data": "true", 2020-10-09T17:35:01.168 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.168 INFO:teuthology.orchestra.run.smithi184.stdout: "read_only": "false", 2020-10-09T17:35:01.168 INFO:teuthology.orchestra.run.smithi184.stdout: "tier_type": "", 2020-10-09T17:35:01.168 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.169 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_from": [], 2020-10-09T17:35:01.169 INFO:teuthology.orchestra.run.smithi184.stdout: "redirect_zone": "" 2020-10-09T17:35:01.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.171 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.171 INFO:teuthology.orchestra.run.smithi184.stdout: "placement_targets": [ 2020-10-09T17:35:01.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.172 INFO:teuthology.orchestra.run.smithi184.stdout: "name": "default-placement", 2020-10-09T17:35:01.172 INFO:teuthology.orchestra.run.smithi184.stdout: "tags": [], 2020-10-09T17:35:01.172 INFO:teuthology.orchestra.run.smithi184.stdout: "storage_classes": [ 2020-10-09T17:35:01.172 INFO:teuthology.orchestra.run.smithi184.stdout: "STANDARD" 2020-10-09T17:35:01.173 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:01.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.173 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "default_placement": "default-placement", 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_policy": { 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: "groups": [] 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: ], 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "short_zone_ids": [ 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 530799950 2020-10-09T17:35:01.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1014881947 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 2021681268 2020-10-09T17:35:01.176 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 1521975245 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": "d44f7e25-d35f-4de0-8a3b-c9248a88bcf5", 2020-10-09T17:35:01.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": 241095183 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: "period_config": { 2020-10-09T17:35:01.178 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket_quota": { 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:35:01.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "user_quota": { 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "enabled": false, 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "check_on_raw": false, 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size": -1, 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "max_size_kb": 0, 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: "max_objects": -1 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:01.180 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:01.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_name": "test-realm", 2020-10-09T17:35:01.181 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:01.181 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:01.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.183 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.226 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:01.242+0000 7fcdd17ef0c0 0 failed reading obj info from .rgw.root:zone_info.879c40c2-988f-403b-a66c-6906292d3ac2: (2) No such file or directory 2020-10-09T17:35:01.248 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:01.242+0000 7fcdd17ef0c0 0 WARNING: could not read zone params for zone id=879c40c2-988f-403b-a66c-6906292d3ac2 name=test-zone4 2020-10-09T17:35:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:01.242+0000 7fcdd17ef0c0 0 failed reading obj info from .rgw.root:zone_info.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84: (2) No such file or directory 2020-10-09T17:35:01.249 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:01.242+0000 7fcdd17ef0c0 0 WARNING: could not read zone params for zone id=bf2453fb-36ea-45e0-8abf-a48b7c4b1b84 name=test-zone2 2020-10-09T17:35:01.256 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:01.256 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:35:01.256 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:35:01.256 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:35:01.256 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.257 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.258 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.259 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone4", 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.260 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "pubsub", 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "false", 2020-10-09T17:35:01.261 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [ 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: "test-zone1" 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:01.262 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.263 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.264 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.265 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.266 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:01.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.268 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:01.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.270 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.352 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.353 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.373 INFO:teuthology.orchestra.run.smithi060.stderr:zone name remove not found in zonegroup test-zonegroup 2020-10-09T17:35:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-10-09T17:35:01.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.381 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.417 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.418 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.457 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:35:01.454+0000 7f0d0d67b0c0 0 failed reading obj info from .rgw.root:zone_info.79f51641-68d6-47cf-b042-61da80f61d5b: (2) No such file or directory 2020-10-09T17:35:01.457 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:35:01.454+0000 7f0d0d67b0c0 0 WARNING: could not read zone params for zone id=79f51641-68d6-47cf-b042-61da80f61d5b name=test-zone3 2020-10-09T17:35:01.458 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:35:01.454+0000 7f0d0d67b0c0 0 failed reading obj info from .rgw.root:zone_info.cc444700-5757-461c-a1a0-0c413b7ba0e8: (2) No such file or directory 2020-10-09T17:35:01.458 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:35:01.454+0000 7f0d0d67b0c0 0 WARNING: could not read zone params for zone id=cc444700-5757-461c-a1a0-0c413b7ba0e8 name=test-zone1 2020-10-09T17:35:01.466 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:35:01.462+0000 7f0d0d67b0c0 0 zone id d44f7e25-d35f-4de0-8a3b-c9248a88bcf5 is not a part of zonegroup default 2020-10-09T17:35:01.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.481 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.526 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.526 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.597 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:01.594+0000 7fc81f7870c0 0 WARNING: can't generate connection for zone d44f7e25-d35f-4de0-8a3b-c9248a88bcf5 id remove: no endpoints defined 2020-10-09T17:35:01.783 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 7, 2020-10-09T17:35:01.783 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [ 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.784 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:35:01.785 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.786 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.787 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone4", 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.789 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "pubsub", 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "false", 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [ 2020-10-09T17:35:01.790 INFO:teuthology.orchestra.run.smithi060.stdout: "test-zone1" 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:35:01.791 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.792 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:35:01.793 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:01.794 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.795 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.796 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.797 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:01.808 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:35:01.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.808 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 530799950 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1014881947 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 2021681268 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:35:01.810 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:35:01.811 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:01.812 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.813 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:35:01.814 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:35:01.814 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:01.815 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-10-09T17:35:01.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.816 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.858 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.859 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:01.882 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:01.882 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "2b195451-6531-416e-b18c-4e330ff14e12", 2020-10-09T17:35:01.882 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "remove_zg", 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "remove_zg", 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "false", 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [], 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:01.883 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "", 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [], 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [], 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "", 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:01.884 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:01.885 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:01.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:01.887 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:01.968 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:02.027 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:02.027 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "c7568d7c-f3eb-47bd-bc7d-b658faebcaa7", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "remove", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "domain_root": "remove.rgw.meta:root", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "control_pool": "remove.rgw.control", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "gc_pool": "remove.rgw.log:gc", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "lc_pool": "remove.rgw.log:lc", 2020-10-09T17:35:02.028 INFO:teuthology.orchestra.run.smithi060.stdout: "log_pool": "remove.rgw.log", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "intent_log_pool": "remove.rgw.log:intent", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "usage_log_pool": "remove.rgw.log:usage", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "roles_pool": "remove.rgw.meta:roles", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "reshard_pool": "remove.rgw.log:reshard", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2020-10-09T17:35:02.029 INFO:teuthology.orchestra.run.smithi060.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "otp_pool": "remove.rgw.otp", 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "system_key": { 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "access_key": "", 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: "secret_key": "" 2020-10-09T17:35:02.030 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_pools": [ 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "default-placement", 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "index_pool": "remove.rgw.buckets.index", 2020-10-09T17:35:02.031 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": { 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD": { 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "data_pool": "remove.rgw.buckets.data" 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2020-10-09T17:35:02.032 INFO:teuthology.orchestra.run.smithi060.stdout: "index_type": 0 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout: "notif_pool": "remove.rgw.log:notif" 2020-10-09T17:35:02.033 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:02.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.036 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:02.117 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:02.118 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:02.381 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 8, 2020-10-09T17:35:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:35:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [ 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:35:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "2b195451-6531-416e-b18c-4e330ff14e12", 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "remove_zg", 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "remove_zg", 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "false", 2020-10-09T17:35:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [], 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "c7568d7c-f3eb-47bd-bc7d-b658faebcaa7", 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "c7568d7c-f3eb-47bd-bc7d-b658faebcaa7", 2020-10-09T17:35:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "remove", 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [], 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "false", 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:35:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:35:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:35:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:35:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone4", 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:35:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "pubsub", 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "false", 2020-10-09T17:35:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [ 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "test-zone1" 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:35:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:35:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:35:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:35:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 530799950 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1014881947 2020-10-09T17:35:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 2021681268 2020-10-09T17:35:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "c7568d7c-f3eb-47bd-bc7d-b658faebcaa7", 2020-10-09T17:35:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 4034756076 2020-10-09T17:35:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:35:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:35:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:35:02.423 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:02.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.425 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:02.504 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:02.504 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:02.558 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:02.554+0000 7f7d09d130c0 0 zone id c7568d7c-f3eb-47bd-bc7d-b658faebcaa7 is not a part of zonegroup test-zonegroup 2020-10-09T17:35:02.559 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:02.554+0000 7f7d09d130c0 0 zone id c7568d7c-f3eb-47bd-bc7d-b658faebcaa7 is not a part of zonegroup default 2020-10-09T17:35:02.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.571 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:02.653 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:02.695 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:35:02.690+0000 7fb126a970c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-10-09T17:35:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-10-09T17:35:02.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.703 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:02.783 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:02.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:35:02.815 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:35:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:02.902 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:03.131 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:35:03.132 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:03.132 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 9, 2020-10-09T17:35:03.132 INFO:teuthology.orchestra.run.smithi060.stdout: "predecessor_uuid": "f137cd21-eb16-49e0-9fe5-a2877c0fdad4", 2020-10-09T17:35:03.132 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": [ 2020-10-09T17:35:03.132 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "1_1602264494.984679_341.1" 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "period_map": { 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:03.133 INFO:teuthology.orchestra.run.smithi060.stdout: "zonegroups": [ 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "test-zonegroup", 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zonegroup", 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "api_name": "", 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "is_master": "true", 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:03.134 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames": [], 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "hostnames_s3website": [], 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "zones": [ 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:03.135 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone3", 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8001" 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:03.136 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.137 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone4", 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8001" 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:03.138 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "pubsub", 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "false", 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [ 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "test-zone1" 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:03.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone2", 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi184.front.sepia.ceph.com:8000" 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.140 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:03.141 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: "id": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test-zone1", 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: "endpoints": [ 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: "http://smithi060.front.sepia.ceph.com:8000" 2020-10-09T17:35:03.142 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "log_meta": "false", 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "log_data": "true", 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_index_max_shards": 11, 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "read_only": "false", 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "tier_type": "", 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from_all": "true", 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_from": [], 2020-10-09T17:35:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_zone": "" 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: "placement_targets": [ 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default-placement", 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2020-10-09T17:35:03.144 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_classes": [ 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: "STANDARD" 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: "default_placement": "default-placement", 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:03.145 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_policy": { 2020-10-09T17:35:03.146 INFO:teuthology.orchestra.run.smithi060.stdout: "groups": [] 2020-10-09T17:35:03.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.146 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.146 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2020-10-09T17:35:03.155 INFO:teuthology.orchestra.run.smithi060.stdout: "short_zone_ids": [ 2020-10-09T17:35:03.155 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.155 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 530799950 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "879c40c2-988f-403b-a66c-6906292d3ac2", 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1014881947 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.156 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 2021681268 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 1521975245 2020-10-09T17:35:03.157 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zonegroup": "test-zonegroup", 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: "master_zone": "cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: "period_config": { 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket_quota": { 2020-10-09T17:35:03.158 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "user_quota": { 2020-10-09T17:35:03.159 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": false, 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "check_on_raw": false, 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size": -1, 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "max_size_kb": 0, 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "max_objects": -1 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_id": "24366018-2cc1-4951-82d7-d435fdbed094", 2020-10-09T17:35:03.160 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_name": "test-realm", 2020-10-09T17:35:03.161 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:35:03.161 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:35:03.161 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-10-09T17:35:03.162 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-50 2020-10-09T17:35:03.250 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-51 2020-10-09T17:35:04.165 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-52 2020-10-09T17:35:04.229 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-53 2020-10-09T17:35:04.258 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-54 2020-10-09T17:35:04.325 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-55 2020-10-09T17:35:06.590 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-10-09T17:35:06.591 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-56 2020-10-09T17:35:06.630 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-57 2020-10-09T17:35:06.699 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-58 2020-10-09T17:35:06.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:06.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:06.761 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:06.796 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:07.015 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:35:07.054 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:07.055 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:35:07.056 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:35:07.057 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264906.736604_323.1", 2020-10-09T17:35:07.057 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:35:06.736604Z" 2020-10-09T17:35:07.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:35:07.057 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:35:07.057 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:35:07.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:07.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:07.059 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:07.091 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:07.092 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:07.335 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:07.335 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:07.335 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:07.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:07.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:07.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:07.338 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:07.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:07.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:07.342 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.343 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:07.344 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:07.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:12.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:12.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:12.347 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:12.388 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:12.389 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:12.653 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:12.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:12.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:12.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:12.658 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:12.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:12.660 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:12.662 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:12.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:17.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:17.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:17.666 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:17.706 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:17.706 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:17.987 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:17.987 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:17.987 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:17.987 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:17.987 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:17.988 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:17.989 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:17.990 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.991 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:17.992 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:17.993 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:17.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:17.995 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:17.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:17.996 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:17.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:22.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:22.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:23.000 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:23.041 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:23.041 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:23.372 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:23.373 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:23.373 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:23.373 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:23.373 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:23.373 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:23.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:23.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:23.374 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:23.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:23.374 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:23.375 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:23.376 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:23.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:23.378 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:23.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:23.381 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:23.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:23.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:23.383 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:28.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:28.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:28.386 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:28.425 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:28.425 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:28.693 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:28.693 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:28.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:28.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:28.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:28.697 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:28.698 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:28.700 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:28.701 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:28.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:28.702 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:28.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:28.702 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:28.703 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:33.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:33.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:33.705 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:33.745 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:33.746 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:34.013 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:34.013 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:34.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:34.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:34.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:34.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:34.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:34.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:34.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:34.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:34.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:34.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:34.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:34.022 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:34.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:34.023 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:34.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:39.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:39.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:39.026 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:39.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:39.067 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:39.322 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:39.331 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:39.332 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:44.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:44.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:44.335 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:44.376 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:44.376 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:44.638 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:44.638 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:44.638 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:44.639 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:44.639 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:44.639 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:44.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:44.640 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:44.641 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:44.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:44.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:44.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:44.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:44.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:44.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:44.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:44.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:44.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:44.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:44.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:44.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:44.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:44.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:44.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:44.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.650 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:44.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:44.650 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:44.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:44.651 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:44.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:44.651 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:44.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:49.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:49.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:49.654 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:49.695 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:49.956 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:49.956 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:49.956 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:49.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:49.958 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:49.958 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:49.958 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:49.958 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:49.958 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.959 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:49.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:49.961 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:49.962 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:49.963 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:49.965 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:49.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:35:54.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:54.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:35:54.970 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:35:55.011 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:35:55.011 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:35:55.277 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:35:55.395 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:35:55.396 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:35:55.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:35:55.396 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:35:55.396 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:35:55.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:55.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:35:55.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:55.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:55.400 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:35:55.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:55.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:55.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:55.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:35:55.401 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:55.402 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:55.402 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:35:55.402 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:55.402 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:55.402 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:35:55.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:35:55.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:35:55.405 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:35:55.406 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:35:55.406 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:35:55.407 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:35:55.407 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:35:55.407 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:35:55.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:00.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:00.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:00.411 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:00.453 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:00.453 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:00.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:00.725 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:00.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:00.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:00.728 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:00.729 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:00.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:00.731 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:00.732 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:00.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:00.734 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:00.734 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:05.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:05.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:05.737 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:05.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:05.779 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:06.050 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:06.050 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:06.050 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:06.050 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:06.050 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:06.051 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:06.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:06.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:06.053 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:06.053 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:06.053 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:06.054 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:06.054 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:06.054 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:06.054 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:06.060 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:06.061 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:11.063 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:11.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:11.064 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:11.104 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:11.105 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:11.376 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:11.378 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.379 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:11.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:11.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:11.382 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:11.383 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:11.385 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:11.386 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:16.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:16.388 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:16.389 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:16.429 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:16.429 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:16.707 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:16.708 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:16.708 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:16.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:16.708 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:16.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:16.710 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:16.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:16.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:16.714 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:16.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:16.716 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:16.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:16.717 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:16.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:21.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:21.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:21.720 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:21.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:21.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:22.027 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:22.027 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:22.027 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:22.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:22.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:22.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:22.030 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:22.031 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:22.032 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:22.034 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:22.035 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:22.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:22.036 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:22.037 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:27.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:27.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:27.040 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:27.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:27.080 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:27.345 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:27.345 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:27.345 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:27.345 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:27.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:27.347 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:27.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:27.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:27.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:27.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:27.349 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:27.349 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:27.349 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:27.349 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:27.349 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:27.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:27.350 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:27.350 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:27.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:27.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:27.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:27.351 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:27.352 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:27.352 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:27.352 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:27.352 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:27.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:27.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:27.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:27.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:27.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:27.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:27.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:27.355 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:27.355 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:27.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:27.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:27.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:27.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:27.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:27.357 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:27.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:27.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:27.358 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:27.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:27.358 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:27.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:27.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:27.360 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:32.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:32.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:32.363 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:32.406 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:32.407 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:32.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:32.684 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:32.685 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:32.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:32.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:32.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:32.689 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:32.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:32.741 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:32.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:37.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:37.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:37.745 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:37.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:37.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:38.062 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:38.062 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:38.063 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:38.064 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:38.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:38.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:38.067 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:38.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:38.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:38.071 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:38.072 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:43.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:43.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:43.075 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:43.117 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:43.118 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:43.369 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:43.370 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:43.370 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:43.371 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:43.371 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:43.371 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:43.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:43.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:43.372 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:43.372 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:43.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:43.373 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:43.373 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:43.373 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:43.373 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:43.374 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:43.374 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:43.374 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:43.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:43.375 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:43.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:43.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:43.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:43.376 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:43.376 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:43.376 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:43.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:43.377 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:43.377 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:43.377 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:43.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:43.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:43.378 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:43.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:43.379 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:43.379 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:43.380 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:43.380 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:43.380 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:43.380 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:43.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:43.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:43.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:43.384 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:43.385 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:43.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:48.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:48.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:48.388 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:48.428 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:48.428 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:48.703 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:48.703 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:48.703 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:48.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:48.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:48.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:48.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:48.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:48.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:48.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:48.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:48.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:48.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:48.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:48.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:48.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:48.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:48.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:48.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:48.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:48.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:48.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:48.711 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:48.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:48.711 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:48.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:48.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:48.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:48.712 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:48.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:48.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:48.714 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:48.714 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:48.714 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:48.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:48.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.715 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:48.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:48.717 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:53.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:53.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:53.719 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:53.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:53.762 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:54.027 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:54.027 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:54.028 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:54.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:54.030 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:54.031 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:54.032 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:54.033 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:54.033 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:54.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:54.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:54.034 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:54.035 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:54.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:54.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:54.038 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:54.039 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:36:59.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:59.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:36:59.042 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:36:59.083 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:36:59.083 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:36:59.339 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:36:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:36:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:36:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:36:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:36:59.340 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:36:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:36:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:36:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:36:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:36:59.349 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:36:59.350 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:37:04.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:04.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:04.352 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:04.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:04.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:04.654 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:04.654 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:04.655 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:04.655 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:04.655 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:04.656 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:37:04.656 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:04.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:04.656 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:04.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:04.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:04.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:04.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:04.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:37:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:04.661 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:37:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:04.662 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:04.663 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264644.317066_210.1", 2020-10-09T17:37:04.664 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:04.665 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:04.665 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:04.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:30:44.317066Z", 2020-10-09T17:37:04.665 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:04.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.666 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.666 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:04.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:04.667 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:04.667 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:37:04.667 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:37:04.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:04.668 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:04.669 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1602264906.736604_323.1 target=1_1602264644.317066_210.1 2020-10-09T17:37:09.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:09.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:09.672 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:09.712 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:09.713 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:09.994 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:09.994 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:09.994 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:09.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:09.995 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:37:09.996 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:09.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:37:09.998 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:09.999 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:10.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.001 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264906.736604_323.1", 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:35:06.736604Z", 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.003 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:10.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:37:10.005 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:37:10.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:10.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:10.006 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:10.046 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:10.047 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:10.302 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:37:10.303 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:37:10.303 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:37:10.303 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:37:10.304 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:37:10.304 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:37:10.304 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:37:10.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:10.305 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:37:10.305 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:10.305 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:37:10.306 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:10.306 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:37:10.306 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:37:10.306 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:10.306 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:10.307 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:10.308 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:37:10.309 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:10.310 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:37:10.310 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:37:10.310 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:10.311 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:10.311 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:10.312 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:37:10.312 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:37:10.312 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.312 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:10.313 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:10.313 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:37:10.313 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:10.314 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:37:10.314 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602264906.736604_323.1", 2020-10-09T17:37:10.314 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:10.315 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:10.315 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:10.315 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:35:06.736604Z", 2020-10-09T17:37:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:37:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:10.317 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:37:10.318 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:37:10.319 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:37:10.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:10.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:10.321 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:10.363 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:10.363 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:10.730 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:37:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602264906.736604_323.1", 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:35:06.736604Z", 2020-10-09T17:37:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:37:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:37:10.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:10.741 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-10-09T17:37:10.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wypfiu-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:10.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wypfiu-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:10.742 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wypfiu-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:37:10.784 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:10.784 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:11.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wypfiu-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:11.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wypfiu-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:11.119 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wypfiu-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:37:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:11.200 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:11.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'wypfiu-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:11.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'wypfiu-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-10-09T17:37:11.528 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket wypfiu-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-10-09T17:37:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:11.610 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:11.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:11.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:11.917 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:11.998 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:11.999 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:12.286 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:37:12.286 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000078", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.history", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000070", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000104", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000026", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20-wypfiu-20", 2020-10-09T17:37:12.287 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21-wypfiu-21", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.9-wypfiu-58", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000028", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.1", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.0.D8yGg12mb7X.0", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:12.288 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8-wypfiu-8", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000040", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.0", 2020-10-09T17:37:12.289 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000015", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000069", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000095", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000003", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000047", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:12.290 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000035", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6-wypfiu-6", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000037", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.4-wypfiu-53", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000024", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000000", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000031", 2020-10-09T17:37:12.291 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.3", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:12.292 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.8-wypfiu-57", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1-wypfiu-1", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000076", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000113", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000005", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000011", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16-wypfiu-16", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000058", 2020-10-09T17:37:12.293 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29-wypfiu-29", 2020-10-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.2", 2020-10-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000052", 2020-10-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18-wypfiu-18", 2020-10-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19-wypfiu-19", 2020-10-09T17:37:12.294 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000088", 2020-10-09T17:37:12.295 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:12.295 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000080", 2020-10-09T17:37:12.295 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000090", 2020-10-09T17:37:12.295 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:12.295 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000110", 2020-10-09T17:37:12.296 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:12.296 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4-wypfiu-47", 2020-10-09T17:37:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000096", 2020-10-09T17:37:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000087", 2020-10-09T17:37:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000008", 2020-10-09T17:37:12.297 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000006", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000029", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39-wypfiu-39", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.3", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000089", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000022", 2020-10-09T17:37:12.298 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13-wypfiu-13", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "bilog.trim", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000016", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:12.299 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000041", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1-wypfiu-49", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000018", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000092", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28-wypfiu-28", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:12.300 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5-wypfiu-5", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000014", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41-wypfiu-41", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25-wypfiu-25", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000112", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5-wypfiu-48", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000007", 2020-10-09T17:37:12.301 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000021", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32-wypfiu-32", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000064", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000071", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000074", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000081", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000009", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.2", 2020-10-09T17:37:12.302 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000121", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.1", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000125", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000082", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.1", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000105", 2020-10-09T17:37:12.303 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000059", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.1.LA/iUW9e4dg.0", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000077", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000032", 2020-10-09T17:37:12.304 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.27-wypfiu-27", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000053", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000091", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000065", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000083", 2020-10-09T17:37:12.305 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000010", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.3", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.3.74nyUmfNlhN.0", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.1", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26-wypfiu-26", 2020-10-09T17:37:12.306 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.2", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37-wypfiu-37", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000045", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000002", 2020-10-09T17:37:12.307 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.5-wypfiu-54", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.2-wypfiu-51", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000116", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10-wypfiu-10", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000034", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14-wypfiu-14", 2020-10-09T17:37:12.308 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000101", 2020-10-09T17:37:12.309 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.1", 2020-10-09T17:37:12.309 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:12.309 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23-wypfiu-23", 2020-10-09T17:37:12.309 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.3", 2020-10-09T17:37:12.310 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000079", 2020-10-09T17:37:12.310 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000049", 2020-10-09T17:37:12.310 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000117", 2020-10-09T17:37:12.310 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000044", 2020-10-09T17:37:12.311 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000066", 2020-10-09T17:37:12.311 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000068", 2020-10-09T17:37:12.311 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000085", 2020-10-09T17:37:12.311 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:12.312 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000073", 2020-10-09T17:37:12.312 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36-wypfiu-36", 2020-10-09T17:37:12.312 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000038", 2020-10-09T17:37:12.312 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.3", 2020-10-09T17:37:12.313 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000118", 2020-10-09T17:37:12.313 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000036", 2020-10-09T17:37:12.313 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.2", 2020-10-09T17:37:12.314 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2-wypfiu-43", 2020-10-09T17:37:12.314 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000103", 2020-10-09T17:37:12.314 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.3", 2020-10-09T17:37:12.315 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.0", 2020-10-09T17:37:12.315 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000119", 2020-10-09T17:37:12.315 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15-wypfiu-15", 2020-10-09T17:37:12.315 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000098", 2020-10-09T17:37:12.316 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22-wypfiu-22", 2020-10-09T17:37:12.316 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.3", 2020-10-09T17:37:12.316 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:12.316 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.0", 2020-10-09T17:37:12.317 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:12.317 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000027", 2020-10-09T17:37:12.317 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12-wypfiu-12", 2020-10-09T17:37:12.318 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status", 2020-10-09T17:37:12.318 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000019", 2020-10-09T17:37:12.318 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:12.318 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:12.319 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000039", 2020-10-09T17:37:12.319 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000100", 2020-10-09T17:37:12.319 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1-wypfiu-44", 2020-10-09T17:37:12.319 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000093", 2020-10-09T17:37:12.320 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000004", 2020-10-09T17:37:12.320 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000063", 2020-10-09T17:37:12.320 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000122", 2020-10-09T17:37:12.321 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000057", 2020-10-09T17:37:12.321 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:12.321 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000054", 2020-10-09T17:37:12.321 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000114", 2020-10-09T17:37:12.322 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:12.322 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.0", 2020-10-09T17:37:12.322 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000012", 2020-10-09T17:37:12.323 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.1", 2020-10-09T17:37:12.323 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.7-wypfiu-56", 2020-10-09T17:37:12.323 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:12.323 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000084", 2020-10-09T17:37:12.324 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000043", 2020-10-09T17:37:12.324 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2-wypfiu-2", 2020-10-09T17:37:12.324 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:12.324 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:12.325 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:12.325 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000111", 2020-10-09T17:37:12.325 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000048", 2020-10-09T17:37:12.325 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:12.326 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000020", 2020-10-09T17:37:12.326 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:37:12.326 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:12.327 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:12.327 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40-wypfiu-40", 2020-10-09T17:37:12.327 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000099", 2020-10-09T17:37:12.327 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000056", 2020-10-09T17:37:12.328 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:12.328 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000072", 2020-10-09T17:37:12.328 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:12.328 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000062", 2020-10-09T17:37:12.329 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:12.329 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000115", 2020-10-09T17:37:12.329 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000033", 2020-10-09T17:37:12.330 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:12.330 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:12.330 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000025", 2020-10-09T17:37:12.330 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000123", 2020-10-09T17:37:12.331 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000108", 2020-10-09T17:37:12.331 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000094", 2020-10-09T17:37:12.331 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:12.332 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24-wypfiu-24", 2020-10-09T17:37:12.332 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4-wypfiu-4", 2020-10-09T17:37:12.332 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000060", 2020-10-09T17:37:12.332 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000109", 2020-10-09T17:37:12.333 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9-wypfiu-9", 2020-10-09T17:37:12.333 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000013", 2020-10-09T17:37:12.333 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.30-wypfiu-30", 2020-10-09T17:37:12.334 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3-wypfiu-3", 2020-10-09T17:37:12.334 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:12.334 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1-wypfiu-42", 2020-10-09T17:37:12.334 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:12.335 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000051", 2020-10-09T17:37:12.335 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000106", 2020-10-09T17:37:12.335 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17-wypfiu-17", 2020-10-09T17:37:12.336 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.0", 2020-10-09T17:37:12.336 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000017", 2020-10-09T17:37:12.336 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2-wypfiu-45", 2020-10-09T17:37:12.336 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:12.337 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.0", 2020-10-09T17:37:12.337 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.6-wypfiu-55", 2020-10-09T17:37:12.337 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000023", 2020-10-09T17:37:12.338 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:12.338 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:12.338 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:12.339 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7-wypfiu-7", 2020-10-09T17:37:12.339 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000107", 2020-10-09T17:37:12.339 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000046", 2020-10-09T17:37:12.339 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3-wypfiu-46", 2020-10-09T17:37:12.340 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.1-wypfiu-50", 2020-10-09T17:37:12.340 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.2", 2020-10-09T17:37:12.340 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.1", 2020-10-09T17:37:12.341 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31-wypfiu-31", 2020-10-09T17:37:12.341 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.2.bBeN+FLJWop.0", 2020-10-09T17:37:12.341 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:12.341 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34-wypfiu-34", 2020-10-09T17:37:12.342 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.2", 2020-10-09T17:37:12.342 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000067", 2020-10-09T17:37:12.342 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:12.343 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:12.343 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:12.343 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000086", 2020-10-09T17:37:12.343 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:12.344 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:12.344 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.2", 2020-10-09T17:37:12.344 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:12.344 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:12.345 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:12.345 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:12.345 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35-wypfiu-35", 2020-10-09T17:37:12.345 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38-wypfiu-38", 2020-10-09T17:37:12.346 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.0", 2020-10-09T17:37:12.346 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.3-wypfiu-52", 2020-10-09T17:37:12.346 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:12.347 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:37:12.347 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11-wypfiu-11", 2020-10-09T17:37:12.347 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000050", 2020-10-09T17:37:12.347 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000102", 2020-10-09T17:37:12.348 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:12.348 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:12.348 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000124", 2020-10-09T17:37:12.349 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:12.349 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000001", 2020-10-09T17:37:12.349 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000126", 2020-10-09T17:37:12.349 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000055", 2020-10-09T17:37:12.350 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000061", 2020-10-09T17:37:12.350 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000097", 2020-10-09T17:37:12.350 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.33-wypfiu-33", 2020-10-09T17:37:12.351 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.e380f3c0-dc07-4443-880b-21f94e64f4aa.3", 2020-10-09T17:37:12.351 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000042", 2020-10-09T17:37:12.351 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000075", 2020-10-09T17:37:12.351 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000120", 2020-10-09T17:37:12.351 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000030", 2020-10-09T17:37:12.352 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1" 2020-10-09T17:37:12.352 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:37:12.352 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:37:12.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:12.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:12.354 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:12.392 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:12.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:12.637 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.637 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.638 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.639 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.639 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.639 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.641 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):621 BONG BONG BONG. Canceled! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.641 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):638 Someone beat us to it, but we still have to process the journal: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.642 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.642 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.642 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.642 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.643 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.643 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.643 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.644 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.644 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{3}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.646 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):546BONG! BONG! BONG! Canceled! 2020-10-09T17:37:12.646 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.646 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.646 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=1 canceled=1 2020-10-09T17:37:12.647 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.647 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.647 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.647 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.647 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.648 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.643+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{4}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.649 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.649 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.651 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.652 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.652 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.652 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.652 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.653 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.653 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.653 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.654 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):613BONG! BONG! BONG! update failed! info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}}r=-22 2020-10-09T17:37:12.654 INFO:tasks.rgw.c2.client.1.smithi184.stdout:*** Caught signal (Aborted) ** 2020-10-09T17:37:12.655 INFO:tasks.rgw.c2.client.1.smithi184.stdout: in thread 7f1fb9feb700 thread_name:rados_async 2020-10-09T17:37:12.655 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{3}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: 6Kjwy2klsIpqozUz, tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{}} 2020-10-09T17:37:12.658 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.658 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.658 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.658 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.658 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.659 INFO:tasks.rgw.c2.client.1.smithi184.stdout: ceph version 16.0.0-6241-g4b457b38 (4b457b3813ec2da916f501ce132355a10e79330a) pacific (dev) 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f1fff6828a0] 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2: gsignal() 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 3: abort() 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 4: (rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield)+0xec2) [0x7f20001a51b2] 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 5: (rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield)+0x56a) [0x7f20001a5a9a] 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 6: (rgw::cls::fifo::FIFO::push(std::vector > const&, optional_yield)+0x93f) [0x7f20001a67df] 2020-10-09T17:37:12.660 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 7: (rgw::cls::fifo::FIFO::push(ceph::buffer::v15_2_0::list const&, optional_yield)+0x137) [0x7f20001a6ef7] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 8: (RGWDataChangesFIFO::push(int, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&&)+0x41) [0x7f200019a361] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 9: (RGWDataChangesLog::add_entry(RGWBucketInfo const&, int)+0x712) [0x7f200018f462] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 10: (RGWSI_BucketIndex_RADOS::handle_overwrite(RGWBucketInfo const&, RGWBucketInfo const&)+0xaa) [0x7f20001c40ea] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 11: (RGWSI_Bucket_SObj::store_bucket_instance_info(ptr_wrapper&, std::__cxx11::basic_string, std::allocator > const&, RGWBucketInfo&, std::optional, bool, std::chrono::time_point > >, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield)+0x3f1) [0x7f20001cde81] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 12: (RGWMetadataHandlerPut_BucketInstance::put_checked()+0x7f) [0x7f1fffd8417f] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 13: (RGWMetadataHandler_GenericMetaBE::do_put_operate(RGWMetadataHandler_GenericMetaBE::Put*)+0x21) [0x7f1fffe555c1] 2020-10-09T17:37:12.661 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 14: (RGWBucketInstanceMetadataHandler::do_put(RGWSI_MetaBackend_Handler::Op*, std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0x96) [0x7f1fffd84586] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 15: /usr/lib/libradosgw.so.2(+0x5c5f95) [0x7f1fffe54f95] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 16: /usr/lib/libradosgw.so.2(+0x956b3a) [0x7f20001e5b3a] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 17: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f20001e92a3] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 18: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f20001e5de1] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 19: (RGWMetadataHandler_GenericMetaBE::put(std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0xe4) [0x7f1fffe56044] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 20: (RGWMetadataManager::put(std::__cxx11::basic_string, std::allocator >&, ceph::buffer::v15_2_0::list&, optional_yield, RGWMDLogSyncType, obj_version*)+0x2cd) [0x7f1fffe5ac1d] 2020-10-09T17:37:12.662 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 21: (RGWAsyncMetaStoreEntry::_send_request()+0x4c) [0x7f1fffe899dc] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 22: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f1ffff7fcb0] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 23: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f1ffff8885d] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 24: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f1ff56233f2] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 25: (ThreadPool::WorkThread::entry()+0x11) [0x7f1ff56242e1] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 26: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f1fff6776db] 2020-10-09T17:37:12.663 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 27: clone() 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fb9feb700 -1 *** Caught signal (Aborted) ** 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout: in thread 7f1fb9feb700 thread_name:rados_async 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout: ceph version 16.0.0-6241-g4b457b38 (4b457b3813ec2da916f501ce132355a10e79330a) pacific (dev) 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f1fff6828a0] 2020-10-09T17:37:12.664 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2: gsignal() 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 3: abort() 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 4: (rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield)+0xec2) [0x7f20001a51b2] 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 5: (rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield)+0x56a) [0x7f20001a5a9a] 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 6: (rgw::cls::fifo::FIFO::push(std::vector > const&, optional_yield)+0x93f) [0x7f20001a67df] 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 7: (rgw::cls::fifo::FIFO::push(ceph::buffer::v15_2_0::list const&, optional_yield)+0x137) [0x7f20001a6ef7] 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 8: (RGWDataChangesFIFO::push(int, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&&)+0x41) [0x7f200019a361] 2020-10-09T17:37:12.665 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 9: (RGWDataChangesLog::add_entry(RGWBucketInfo const&, int)+0x712) [0x7f200018f462] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 10: (RGWSI_BucketIndex_RADOS::handle_overwrite(RGWBucketInfo const&, RGWBucketInfo const&)+0xaa) [0x7f20001c40ea] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 11: (RGWSI_Bucket_SObj::store_bucket_instance_info(ptr_wrapper&, std::__cxx11::basic_string, std::allocator > const&, RGWBucketInfo&, std::optional, bool, std::chrono::time_point > >, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield)+0x3f1) [0x7f20001cde81] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 12: (RGWMetadataHandlerPut_BucketInstance::put_checked()+0x7f) [0x7f1fffd8417f] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 13: (RGWMetadataHandler_GenericMetaBE::do_put_operate(RGWMetadataHandler_GenericMetaBE::Put*)+0x21) [0x7f1fffe555c1] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 14: (RGWBucketInstanceMetadataHandler::do_put(RGWSI_MetaBackend_Handler::Op*, std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0x96) [0x7f1fffd84586] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 15: /usr/lib/libradosgw.so.2(+0x5c5f95) [0x7f1fffe54f95] 2020-10-09T17:37:12.666 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 16: /usr/lib/libradosgw.so.2(+0x956b3a) [0x7f20001e5b3a] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 17: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f20001e92a3] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 18: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f20001e5de1] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 19: (RGWMetadataHandler_GenericMetaBE::put(std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0xe4) [0x7f1fffe56044] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 20: (RGWMetadataManager::put(std::__cxx11::basic_string, std::allocator >&, ceph::buffer::v15_2_0::list&, optional_yield, RGWMDLogSyncType, obj_version*)+0x2cd) [0x7f1fffe5ac1d] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 21: (RGWAsyncMetaStoreEntry::_send_request()+0x4c) [0x7f1fffe899dc] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 22: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f1ffff7fcb0] 2020-10-09T17:37:12.667 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 23: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f1ffff8885d] 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 24: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f1ff56233f2] 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 25: (ThreadPool::WorkThread::entry()+0x11) [0x7f1ff56242e1] 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 26: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f1fff6776db] 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 27: clone() 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-10-09T17:37:12.668 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.681 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -192> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.681 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -190> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.681 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -189> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.681 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -187> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -186> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -176> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -175> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -174> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -173> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -172> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.682 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -171> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -170> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -169> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):621 BONG BONG BONG. Canceled! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -168> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):638 Someone beat us to it, but we still have to process the journal: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -167> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -166> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -165> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.683 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -164> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -163> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -162> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -161> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -160> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -159> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{3}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -158> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):546BONG! BONG! BONG! Canceled! 2020-10-09T17:37:12.684 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -157> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -156> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -155> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=1 canceled=1 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -139> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -138> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -137> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -136> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.685 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -135> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -134> 2020-10-09T17:37:12.643+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{4}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -133> 2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -132> 2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -84> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -83> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -82> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.686 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -81> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -80> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -25> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -24> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -23> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -22> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):613BONG! BONG! BONG! update failed! info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}}r=-22 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -21> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{3}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: 6Kjwy2klsIpqozUz, tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{}} 2020-10-09T17:37:12.687 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -5> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -4> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -3> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -2> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -1> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 0> 2020-10-09T17:37:12.655+0000 7f1fb9feb700 -1 *** Caught signal (Aborted) ** 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: in thread 7f1fb9feb700 thread_name:rados_async 2020-10-09T17:37:12.688 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: ceph version 16.0.0-6241-g4b457b38 (4b457b3813ec2da916f501ce132355a10e79330a) pacific (dev) 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f1fff6828a0] 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2: gsignal() 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 3: abort() 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 4: (rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield)+0xec2) [0x7f20001a51b2] 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 5: (rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield)+0x56a) [0x7f20001a5a9a] 2020-10-09T17:37:12.689 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 6: (rgw::cls::fifo::FIFO::push(std::vector > const&, optional_yield)+0x93f) [0x7f20001a67df] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 7: (rgw::cls::fifo::FIFO::push(ceph::buffer::v15_2_0::list const&, optional_yield)+0x137) [0x7f20001a6ef7] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 8: (RGWDataChangesFIFO::push(int, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&&)+0x41) [0x7f200019a361] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 9: (RGWDataChangesLog::add_entry(RGWBucketInfo const&, int)+0x712) [0x7f200018f462] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 10: (RGWSI_BucketIndex_RADOS::handle_overwrite(RGWBucketInfo const&, RGWBucketInfo const&)+0xaa) [0x7f20001c40ea] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 11: (RGWSI_Bucket_SObj::store_bucket_instance_info(ptr_wrapper&, std::__cxx11::basic_string, std::allocator > const&, RGWBucketInfo&, std::optional, bool, std::chrono::time_point > >, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield)+0x3f1) [0x7f20001cde81] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 12: (RGWMetadataHandlerPut_BucketInstance::put_checked()+0x7f) [0x7f1fffd8417f] 2020-10-09T17:37:12.690 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 13: (RGWMetadataHandler_GenericMetaBE::do_put_operate(RGWMetadataHandler_GenericMetaBE::Put*)+0x21) [0x7f1fffe555c1] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 14: (RGWBucketInstanceMetadataHandler::do_put(RGWSI_MetaBackend_Handler::Op*, std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0x96) [0x7f1fffd84586] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 15: /usr/lib/libradosgw.so.2(+0x5c5f95) [0x7f1fffe54f95] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 16: /usr/lib/libradosgw.so.2(+0x956b3a) [0x7f20001e5b3a] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 17: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f20001e92a3] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 18: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f20001e5de1] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 19: (RGWMetadataHandler_GenericMetaBE::put(std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0xe4) [0x7f1fffe56044] 2020-10-09T17:37:12.691 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 20: (RGWMetadataManager::put(std::__cxx11::basic_string, std::allocator >&, ceph::buffer::v15_2_0::list&, optional_yield, RGWMDLogSyncType, obj_version*)+0x2cd) [0x7f1fffe5ac1d] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 21: (RGWAsyncMetaStoreEntry::_send_request()+0x4c) [0x7f1fffe899dc] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 22: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f1ffff7fcb0] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 23: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f1ffff8885d] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 24: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f1ff56233f2] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 25: (ThreadPool::WorkThread::entry()+0x11) [0x7f1ff56242e1] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 26: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f1fff6776db] 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 27: clone() 2020-10-09T17:37:12.692 INFO:tasks.rgw.c2.client.1.smithi184.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-10-09T17:37:12.693 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.693 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.0, version: 6ySiDYQ6AfNN48MV{2}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=blxSgP4Q7U4iejP8}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: blxSgP4Q7U4iejP8,0=op: Op::set_head, part_num: 0, part_tag: blxSgP4Q7U4iejP8}} 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: blxSgP4Q7U4iejP8 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: blxSgP4Q7U4iejP8 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.663+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.0, version: 6ySiDYQ6AfNN48MV{3}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: blxSgP4Q7U4iejP8, tags: {{0=blxSgP4Q7U4iejP8}}, journal: {{{}} 2020-10-09T17:37:12.694 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.1, version: PbEQ6Q8+5DMb8lFO{2}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=kmlzkdfADxTWjJRY}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: kmlzkdfADxTWjJRY,0=op: Op::set_head, part_num: 0, part_tag: kmlzkdfADxTWjJRY}} 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: kmlzkdfADxTWjJRY 2020-10-09T17:37:12.695 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: kmlzkdfADxTWjJRY 2020-10-09T17:37:12.696 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.696 INFO:tasks.rgw.c2.client.1.smithi184.stdout:2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.700 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000078", 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.history", 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000070", 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000104", 2020-10-09T17:37:12.701 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000026", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20-wypfiu-20", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21-wypfiu-21", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000028", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:12.702 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3.PfFOjc/IWBR.0", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8-wypfiu-8", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000040", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.0", 2020-10-09T17:37:12.703 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000015", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000069", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000095", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000003", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000047", 2020-10-09T17:37:12.704 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000035", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2.KEgxCJYxQjD.0", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6-wypfiu-6", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000037", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000024", 2020-10-09T17:37:12.705 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000000", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000031", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:12.706 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.8-wypfiu-57", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1-wypfiu-1", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000076", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000113", 2020-10-09T17:37:12.707 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000005", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000011", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16-wypfiu-16", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000058", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000052", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18-wypfiu-18", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19-wypfiu-19", 2020-10-09T17:37:12.708 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000088", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000080", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000090", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000110", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4-wypfiu-47", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:12.709 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000096", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000087", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000008", 2020-10-09T17:37:12.710 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000006", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000029", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39-wypfiu-39", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000089", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000022", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:12.711 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13-wypfiu-13", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "bilog.trim", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000016", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000041", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1-wypfiu-49", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000018", 2020-10-09T17:37:12.712 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000092", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.28-wypfiu-28", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5-wypfiu-5", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000014", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41-wypfiu-41", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25-wypfiu-25", 2020-10-09T17:37:12.713 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000112", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5-wypfiu-48", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000007", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000021", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000064", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000071", 2020-10-09T17:37:12.714 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000074", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000081", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000009", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000121", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1", 2020-10-09T17:37:12.715 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000125", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000082", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.1", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000105", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000059", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1.8/Nh9GjYM0Z.0", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:12.716 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000077", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000032", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000053", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000091", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:12.717 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000065", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000083", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000010", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.1", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26-wypfiu-26", 2020-10-09T17:37:12.718 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0.QOGWW0I4Y/0.0", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.2", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37-wypfiu-37", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000045", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000002", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.5-wypfiu-54", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.2-wypfiu-51", 2020-10-09T17:37:12.719 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000116", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10-wypfiu-10", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000034", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14-wypfiu-14", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000101", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.1", 2020-10-09T17:37:12.720 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23-wypfiu-23", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.3", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000079", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000049", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000117", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000044", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:12.721 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000066", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000068", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000085", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000073", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36-wypfiu-36", 2020-10-09T17:37:12.722 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000038", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.3", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000118", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000036", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.2", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2-wypfiu-43", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000103", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.3", 2020-10-09T17:37:12.723 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000119", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15-wypfiu-15", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000098", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22-wypfiu-22", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.3", 2020-10-09T17:37:12.724 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.0", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000027", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12-wypfiu-12", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status", 2020-10-09T17:37:12.725 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000019", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000039", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000100", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1-wypfiu-44", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000093", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000004", 2020-10-09T17:37:12.726 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000063", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000122", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000057", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000054", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000114", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0", 2020-10-09T17:37:12.727 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000012", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.1", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000084", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000043", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2-wypfiu-2", 2020-10-09T17:37:12.728 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000111", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000048", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:12.729 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000020", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40-wypfiu-40", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000099", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000056", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:12.730 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000072", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000062", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000115", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000033", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:12.731 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000025", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000123", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000108", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000094", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:12.732 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24-wypfiu-24", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4-wypfiu-4", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000060", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000109", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9-wypfiu-9", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000013", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:12.733 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3-wypfiu-3", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1-wypfiu-42", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000051", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000106", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17-wypfiu-17", 2020-10-09T17:37:12.734 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.79f51641-68d6-47cf-b042-61da80f61d5b.0", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000017", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2-wypfiu-45", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000023", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7-wypfiu-7", 2020-10-09T17:37:12.735 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000107", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000046", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3-wypfiu-46", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.2", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.31-wypfiu-31", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:12.736 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.34-wypfiu-34", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000067", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000086", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:12.737 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.2", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38-wypfiu-38", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.79f51641-68d6-47cf-b042-61da80f61d5b.0", 2020-10-09T17:37:12.738 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11-wypfiu-11", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000050", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000102", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:12.739 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000124", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000001", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000126", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000055", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000061", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000097", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.e380f3c0-dc07-4443-880b-21f94e64f4aa.3", 2020-10-09T17:37:12.740 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000042", 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000075", 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000120", 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000030", 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.79f51641-68d6-47cf-b042-61da80f61d5b:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1" 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:37:12.741 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:37:12.745 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -279> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.746 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -277> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.746 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -276> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.746 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -274> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.746 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -273> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.746 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -263> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -262> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -261> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -260> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{1}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -259> 2020-10-09T17:37:12.631+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -258> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -257> 2020-10-09T17:37:12.631+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.747 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -256> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):621 BONG BONG BONG. Canceled! INFO: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -255> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):638 Someone beat us to it, but we still have to process the journal: id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -254> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.2, version: PQfTiKQDUUHlXwy0{2}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD,0=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD}} 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -253> 2020-10-09T17:37:12.635+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -252> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -251> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -250> 2020-10-09T17:37:12.635+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.748 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -249> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: XLuD9wSP9aul1QmD 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -248> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -247> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -246> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{3}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -245> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):546BONG! BONG! BONG! Canceled! 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -244> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -243> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):556BONG! BONG! BONG! Journal entry was already processed 2020-10-09T17:37:12.749 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -242> 2020-10-09T17:37:12.639+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=1 canceled=1 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -226> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -225> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -224> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -223> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{1}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -222> 2020-10-09T17:37:12.639+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -221> 2020-10-09T17:37:12.643+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.2, version: PQfTiKQDUUHlXwy0{4}, oid_prefix: data_log.2.x2Kg+cZFM7P, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: XLuD9wSP9aul1QmD, tags: {{0=XLuD9wSP9aul1QmD}}, journal: {{{}} 2020-10-09T17:37:12.750 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -220> 2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -219> 2020-10-09T17:37:12.643+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -171> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -170> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -169> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -168> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}} 2020-10-09T17:37:12.751 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -167> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -112> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -111> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -110> 2020-10-09T17:37:12.647+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -109> 2020-10-09T17:37:12.647+0000 7f1fb9feb700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):613BONG! BONG! BONG! update failed! info=id: data_log.3, version: cUQSKMfM6tyr+nYT{2}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: 6Kjwy2klsIpqozUz,0=op: Op::set_head, part_num: 0, part_tag: 6Kjwy2klsIpqozUz}}r=-22 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -108> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.3, version: cUQSKMfM6tyr+nYT{3}, oid_prefix: data_log.3.Ax3QZ5tA5NP, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: 6Kjwy2klsIpqozUz, tags: {{0=6Kjwy2klsIpqozUz}}, journal: {{{}} 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -92> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.752 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -91> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -90> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -89> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.0, version: 6ySiDYQ6AfNN48MV{1}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -88> 2020-10-09T17:37:12.651+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -87> 2020-10-09T17:37:12.655+0000 7f1fb9feb700 -1 *** Caught signal (Aborted) ** 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: in thread 7f1fb9feb700 thread_name:rados_async 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.753 INFO:tasks.rgw.c2.client.1.smithi184.stdout: ceph version 16.0.0-6241-g4b457b38 (4b457b3813ec2da916f501ce132355a10e79330a) pacific (dev) 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f1fff6828a0] 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2: gsignal() 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 3: abort() 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 4: (rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield)+0xec2) [0x7f20001a51b2] 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 5: (rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield)+0x56a) [0x7f20001a5a9a] 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 6: (rgw::cls::fifo::FIFO::push(std::vector > const&, optional_yield)+0x93f) [0x7f20001a67df] 2020-10-09T17:37:12.754 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 7: (rgw::cls::fifo::FIFO::push(ceph::buffer::v15_2_0::list const&, optional_yield)+0x137) [0x7f20001a6ef7] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 8: (RGWDataChangesFIFO::push(int, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator > const&, ceph::buffer::v15_2_0::list&&)+0x41) [0x7f200019a361] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 9: (RGWDataChangesLog::add_entry(RGWBucketInfo const&, int)+0x712) [0x7f200018f462] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 10: (RGWSI_BucketIndex_RADOS::handle_overwrite(RGWBucketInfo const&, RGWBucketInfo const&)+0xaa) [0x7f20001c40ea] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 11: (RGWSI_Bucket_SObj::store_bucket_instance_info(ptr_wrapper&, std::__cxx11::basic_string, std::allocator > const&, RGWBucketInfo&, std::optional, bool, std::chrono::time_point > >, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >*, optional_yield)+0x3f1) [0x7f20001cde81] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 12: (RGWMetadataHandlerPut_BucketInstance::put_checked()+0x7f) [0x7f1fffd8417f] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 13: (RGWMetadataHandler_GenericMetaBE::do_put_operate(RGWMetadataHandler_GenericMetaBE::Put*)+0x21) [0x7f1fffe555c1] 2020-10-09T17:37:12.755 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 14: (RGWBucketInstanceMetadataHandler::do_put(RGWSI_MetaBackend_Handler::Op*, std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0x96) [0x7f1fffd84586] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 15: /usr/lib/libradosgw.so.2(+0x5c5f95) [0x7f1fffe54f95] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 16: /usr/lib/libradosgw.so.2(+0x956b3a) [0x7f20001e5b3a] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 17: (RGWSI_MetaBackend_SObj::call(std::optional >, std::function)+0x93) [0x7f20001e92a3] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 18: (RGWSI_MetaBackend_Handler::call(std::optional >, std::function)+0x61) [0x7f20001e5de1] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 19: (RGWMetadataHandler_GenericMetaBE::put(std::__cxx11::basic_string, std::allocator >&, RGWMetadataObject*, RGWObjVersionTracker&, optional_yield, RGWMDLogSyncType)+0xe4) [0x7f1fffe56044] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 20: (RGWMetadataManager::put(std::__cxx11::basic_string, std::allocator >&, ceph::buffer::v15_2_0::list&, optional_yield, RGWMDLogSyncType, obj_version*)+0x2cd) [0x7f1fffe5ac1d] 2020-10-09T17:37:12.756 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 21: (RGWAsyncMetaStoreEntry::_send_request()+0x4c) [0x7f1fffe899dc] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 22: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f1ffff7fcb0] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 23: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f1ffff8885d] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 24: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa52) [0x7f1ff56233f2] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 25: (ThreadPool::WorkThread::entry()+0x11) [0x7f1ff56242e1] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 26: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f1fff6776db] 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 27: clone() 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-10-09T17:37:12.757 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -86> 2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.0, version: 6ySiDYQ6AfNN48MV{2}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=blxSgP4Q7U4iejP8}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: blxSgP4Q7U4iejP8,0=op: Op::set_head, part_num: 0, part_tag: blxSgP4Q7U4iejP8}} 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -85> 2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: blxSgP4Q7U4iejP8 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -84> 2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: blxSgP4Q7U4iejP8 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -83> 2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -82> 2020-10-09T17:37:12.655+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -63> 2020-10-09T17:37:12.663+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):925 BONG BONG BONG: preparing new head! INFO= id: data_log.0, version: 6ySiDYQ6AfNN48MV{3}, oid_prefix: data_log.0.vIILyMPC2QU, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: 0, min_push_part_num: 0, max_push_part_num: 0, head_tag: blxSgP4Q7U4iejP8, tags: {{0=blxSgP4Q7U4iejP8}}, journal: {{{}} 2020-10-09T17:37:12.758 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -45> 2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::push(const std::vector&, optional_yield):908 BONG BONG BONG: need new head. INFO= id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -44> 2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_head(optional_yield):677 BONG BONG BONG. Preparing new part! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -43> 2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):577 BONG BONG BONG. Entering! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -42> 2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):592 BONG BONG BONG. Need to make a New Head! INFO: id: data_log.1, version: PbEQ6Q8+5DMb8lFO{1}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{}}, journal: {{{}} 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -41> 2020-10-09T17:37:12.671+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::_prepare_new_part(bool, optional_yield):606 BONG BONG BONG. Calling update! i=0 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -19> 2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):437BONG! BONG! BONG! Entering process_journal info=id: data_log.1, version: PbEQ6Q8+5DMb8lFO{2}, oid_prefix: data_log.1.a0N45VFEgxI, params: {max_part_size: 4194304, max_entry_size: 32768, full_size_threshold: 4161478}, tail_part_num: 0, head_part_num: -1, min_push_part_num: 0, max_push_part_num: -1, head_tag: , tags: {{0=kmlzkdfADxTWjJRY}}, journal: {{{0=op: Op::create, part_num: 0, part_tag: kmlzkdfADxTWjJRY,0=op: Op::set_head, part_num: 0, part_tag: kmlzkdfADxTWjJRY}} 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -18> 2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::create, part_num: 0, part_tag: kmlzkdfADxTWjJRY 2020-10-09T17:37:12.759 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -2> 2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):451BONG! BONG! BONG! running n=0 entry=op: Op::set_head, part_num: 0, part_tag: kmlzkdfADxTWjJRY 2020-10-09T17:37:12.760 INFO:tasks.rgw.c2.client.1.smithi184.stdout: -1> 2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):497 BONG! BONG! BONG! Postprocessing! 2020-10-09T17:37:12.760 INFO:tasks.rgw.c2.client.1.smithi184.stdout: 0> 2020-10-09T17:37:12.675+0000 7f1fc0ff9700 -1 int rgw::cls::fifo::FIFO::process_journal(optional_yield):506 BONG! BONG! BONG! Postprocessing: i=0 canceled=1 2020-10-09T17:37:12.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:12.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:12.761 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:12.762 INFO:tasks.rgw.c2.client.1.smithi184.stderr:daemon-helper: command crashed with signal 6 2020-10-09T17:37:12.795 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:12.796 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:13.065 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:37:13.098 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000078", 2020-10-09T17:37:13.098 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.history", 2020-10-09T17:37:13.098 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:13.098 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000070", 2020-10-09T17:37:13.099 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000104", 2020-10-09T17:37:13.099 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:13.099 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000026", 2020-10-09T17:37:13.099 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20-wypfiu-20", 2020-10-09T17:37:13.099 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21-wypfiu-21", 2020-10-09T17:37:13.100 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.9-wypfiu-58", 2020-10-09T17:37:13.100 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000028", 2020-10-09T17:37:13.100 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.1", 2020-10-09T17:37:13.100 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:13.100 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:13.101 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.8-wypfiu-8", 2020-10-09T17:37:13.101 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:13.101 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:13.101 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000040", 2020-10-09T17:37:13.101 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.0", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000015", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000069", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000095", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000003", 2020-10-09T17:37:13.102 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000047", 2020-10-09T17:37:13.103 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:13.103 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:13.103 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000035", 2020-10-09T17:37:13.103 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.6-wypfiu-6", 2020-10-09T17:37:13.103 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000037", 2020-10-09T17:37:13.104 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.0", 2020-10-09T17:37:13.104 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:13.104 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.3", 2020-10-09T17:37:13.104 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000024", 2020-10-09T17:37:13.104 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000000", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000031", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.3", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:13.105 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.1-wypfiu-1", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000076", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000113", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000005", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000011", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16-wypfiu-16", 2020-10-09T17:37:13.106 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000058", 2020-10-09T17:37:13.107 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.29-wypfiu-29", 2020-10-09T17:37:13.107 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.2", 2020-10-09T17:37:13.107 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000052", 2020-10-09T17:37:13.107 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:13.107 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18-wypfiu-18", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19-wypfiu-19", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000088", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000080", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000090", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.3.PhglkOPoiz1.0", 2020-10-09T17:37:13.108 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000110", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4-wypfiu-47", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:13.109 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000096", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000087", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:13.110 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000008", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000006", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000029", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.39-wypfiu-39", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.3", 2020-10-09T17:37:13.111 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000089", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000022", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13-wypfiu-13", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "bilog.trim", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1", 2020-10-09T17:37:13.112 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000016", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000041", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1-wypfiu-49", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000018", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000092", 2020-10-09T17:37:13.113 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.5-wypfiu-5", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000014", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.41-wypfiu-41", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25-wypfiu-25", 2020-10-09T17:37:13.114 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000112", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5-wypfiu-48", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000007", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000021", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.32-wypfiu-32", 2020-10-09T17:37:13.115 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000064", 2020-10-09T17:37:13.116 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000071", 2020-10-09T17:37:13.116 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000074", 2020-10-09T17:37:13.116 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:13.116 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:13.116 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000081", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000009", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.2", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000121", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.1", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000125", 2020-10-09T17:37:13.117 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000082", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.1", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000105", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000059", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:13.118 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000077", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000032", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000053", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000091", 2020-10-09T17:37:13.119 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000065", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000083", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000010", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.3", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26-wypfiu-26", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.2", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37-wypfiu-37", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:13.121 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000045", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000002", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000116", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10-wypfiu-10", 2020-10-09T17:37:13.122 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000034", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14-wypfiu-14", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000101", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.1", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:13.123 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23-wypfiu-23", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000079", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000049", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000117", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000044", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:13.124 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000066", 2020-10-09T17:37:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000068", 2020-10-09T17:37:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000085", 2020-10-09T17:37:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000073", 2020-10-09T17:37:13.125 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36-wypfiu-36", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000038", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.3", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000118", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000036", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.2-wypfiu-43", 2020-10-09T17:37:13.126 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000103", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.3", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.0", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000119", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15-wypfiu-15", 2020-10-09T17:37:13.127 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000098", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22-wypfiu-22", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status.shard.0", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000027", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12-wypfiu-12", 2020-10-09T17:37:13.128 INFO:teuthology.orchestra.run.smithi060.stdout: "mdlog.sync-status", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000019", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000039", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:13.129 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.f137cd21-eb16-49e0-9fe5-a2877c0fdad4.3", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000100", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.1-wypfiu-44", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000093", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.1.szQWweS2FIg.0", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000004", 2020-10-09T17:37:13.130 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000063", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000122", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000057", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.2", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000054", 2020-10-09T17:37:13.131 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000114", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.0", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000012", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.2.CSAj2WDgH+1.0", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:13.132 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000084", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000043", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.2-wypfiu-2", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:13.133 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000111", 2020-10-09T17:37:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000048", 2020-10-09T17:37:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000020", 2020-10-09T17:37:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84", 2020-10-09T17:37:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:13.134 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.40-wypfiu-40", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000099", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000056", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000072", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:13.135 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000062", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000115", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000033", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:13.136 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000025", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000123", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000108", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000094", 2020-10-09T17:37:13.137 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24-wypfiu-24", 2020-10-09T17:37:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.4-wypfiu-4", 2020-10-09T17:37:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000060", 2020-10-09T17:37:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000109", 2020-10-09T17:37:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:13.138 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.9-wypfiu-9", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000013", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.3-wypfiu-3", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.5102.1-wypfiu-42", 2020-10-09T17:37:13.139 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000051", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000106", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17-wypfiu-17", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000017", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.2-wypfiu-45", 2020-10-09T17:37:13.140 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.0", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.6-wypfiu-55", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000023", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.7-wypfiu-7", 2020-10-09T17:37:13.141 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000107", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000046", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3-wypfiu-46", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.full-sync.index.2", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "datalog.sync-status.shard.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84.1", 2020-10-09T17:37:13.142 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.2", 2020-10-09T17:37:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000067", 2020-10-09T17:37:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:13.143 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000086", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:13.144 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.35-wypfiu-35", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38-wypfiu-38", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.5475.3-wypfiu-52", 2020-10-09T17:37:13.145 INFO:teuthology.orchestra.run.smithi060.stdout: "2020-10-09-17-cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11-wypfiu-11", 2020-10-09T17:37:13.146 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:13.146 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000050", 2020-10-09T17:37:13.146 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000102", 2020-10-09T17:37:13.146 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:13.146 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000124", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.bf2453fb-36ea-45e0-8abf-a48b7c4b1b84:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000001", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000126", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000055", 2020-10-09T17:37:13.147 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000061", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000097", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "meta.log.e380f3c0-dc07-4443-880b-21f94e64f4aa.3", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000042", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "data_log.0.s2SpMq60as+.0", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000075", 2020-10-09T17:37:13.148 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000120", 2020-10-09T17:37:13.149 INFO:teuthology.orchestra.run.smithi060.stdout: "obj_delete_at_hint.0000000030" 2020-10-09T17:37:13.149 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:37:13.149 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-10-09T17:37:13.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.151 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:13.187 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:13.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:13.464 INFO:teuthology.orchestra.run.smithi184.stdout:[ 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000078", 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.history", 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:3", 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000070", 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000104", 2020-10-09T17:37:13.465 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0.vIILyMPC2QU.0", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000026", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3.Ax3QZ5tA5NP.0", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000028", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:5", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:7", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000040", 2020-10-09T17:37:13.466 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.0", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000015", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000069", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000095", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000003", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000047", 2020-10-09T17:37:13.467 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000035", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000037", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.0", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.3", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000024", 2020-10-09T17:37:13.468 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000000", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000031", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1.a0N45VFEgxI.0", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.3", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:8", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000076", 2020-10-09T17:37:13.469 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000113", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000005", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000011", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000058", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000052", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-24:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.24:8", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000088", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000080", 2020-10-09T17:37:13.470 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000090", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000110", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-21:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.21:8", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.1", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-38:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.38:10", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000096", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:7", 2020-10-09T17:37:13.471 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000087", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-14:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.14:1", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000008", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000006", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000029", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000089", 2020-10-09T17:37:13.472 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000022", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "bilog.trim", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000016", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000041", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-15:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.15:1", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000018", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000092", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:7", 2020-10-09T17:37:13.473 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000014", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000112", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000007", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000021", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-36:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.36:10", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000064", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000071", 2020-10-09T17:37:13.474 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000074", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-47:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.4:8", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-26:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.26:8", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000081", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000009", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000121", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.1", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000125", 2020-10-09T17:37:13.475 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000082", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.1", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000105", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000059", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000077", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000032", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:3", 2020-10-09T17:37:13.476 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:1", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000053", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000091", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000065", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000083", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000010", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-13:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.13:1", 2020-10-09T17:37:13.477 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.2", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000045", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000002", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000116", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:1", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000034", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000101", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2.x2Kg+cZFM7P.0", 2020-10-09T17:37:13.478 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.1", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000079", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000049", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000117", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000044", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:3", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000066", 2020-10-09T17:37:13.479 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000068", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000085", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-20:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.20:8", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000073", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000038", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.3", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000118", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000036", 2020-10-09T17:37:13.480 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000103", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.3", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-23:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.23:8", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000119", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-49:cc444700-5757-461c-a1a0-0c413b7ba0e8.5343.1:5", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000098", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status.shard.0", 2020-10-09T17:37:13.481 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000027", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "mdlog.sync-status", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000019", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000039", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:5", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.f137cd21-eb16-49e0-9fe5-a2877c0fdad4.3", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000100", 2020-10-09T17:37:13.482 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000093", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000004", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000063", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000122", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000057", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.eb32ec64-e1b3-4311-8266-3310720f134e.2", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000054", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000114", 2020-10-09T17:37:13.483 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.0", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000012", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:3", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:8", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000084", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000043", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:1", 2020-10-09T17:37:13.484 INFO:teuthology.orchestra.run.smithi184.stdout: "data.full-sync.index.cc444700-5757-461c-a1a0-0c413b7ba0e8.3", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000111", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000048", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000020", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-48:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.5:8", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000099", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000056", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000072", 2020-10-09T17:37:13.485 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000062", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.2", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000115", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000033", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-17:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.17:5", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-37:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.37:10", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-12:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.12:3", 2020-10-09T17:37:13.486 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000025", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000123", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000108", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000094", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000060", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000109", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-10:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.10:8", 2020-10-09T17:37:13.487 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000013", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "datalog.sync-status.shard.cc444700-5757-461c-a1a0-0c413b7ba0e8.0", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-19:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.19:8", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000051", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000106", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000017", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000023", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000107", 2020-10-09T17:37:13.488 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000046", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-22:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.22:8", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.full-sync.index.2", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "data_log.2", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-46:cc444700-5757-461c-a1a0-0c413b7ba0e8.5143.3:8", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000067", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:1", 2020-10-09T17:37:13.489 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000086", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:3", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-18:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.18:1", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-11:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.11:1", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-25:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.25:8", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000050", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000102", 2020-10-09T17:37:13.490 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000124", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "bucket.sync-status.cc444700-5757-461c-a1a0-0c413b7ba0e8:wypfiu-16:cc444700-5757-461c-a1a0-0c413b7ba0e8.4342.16:5", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000001", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000126", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000055", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000061", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000097", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "meta.log.e380f3c0-dc07-4443-880b-21f94e64f4aa.3", 2020-10-09T17:37:13.491 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000042", 2020-10-09T17:37:13.492 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000075", 2020-10-09T17:37:13.492 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000120", 2020-10-09T17:37:13.492 INFO:teuthology.orchestra.run.smithi184.stdout: "obj_delete_at_hint.0000000030" 2020-10-09T17:37:13.492 INFO:teuthology.orchestra.run.smithi184.stdout:] 2020-10-09T17:37:13.492 INFO:teuthology.orchestra.run.smithi184.stdout: 2020-10-09T17:37:13.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.595 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:13.636 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:13.636 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:13.907 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:37:13.968 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.901248Z" 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-09T17:37:13.969 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.903341Z" 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.905919Z" 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:13.970 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:13.971 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-10-09T17:37:13.971 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.908703Z" 2020-10-09T17:37:13.971 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:13.971 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:37:13.972 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:37:13.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:13.973 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:14.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:14.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:14.270 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:14.270 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:14.271 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:14.272 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:14.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:14.274 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:14.275 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:14.276 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:14.277 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:14.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:14.278 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:14.279 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:16.408 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-10-09T17:37:19.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:19.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:19.282 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:19.325 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:19.325 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:19.590 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:19.590 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:19.591 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:19.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:19.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:19.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:19.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:19.596 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:19.596 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:19.596 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:19.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:19.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:19.597 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:19.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:19.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:19.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:19.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:19.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:19.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:19.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:19.600 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:19.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:19.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.600 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:19.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:19.603 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:22.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2020-10-09T17:37:24.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:24.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:24.606 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:24.648 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:24.649 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:24.914 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:24.914 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:24.914 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:24.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:24.916 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:24.917 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:24.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:24.919 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:24.921 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:24.922 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:24.923 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:24.923 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:24.924 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:27.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~11s 2020-10-09T17:37:29.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:29.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:29.926 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:29.970 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:29.970 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:30.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:30.230 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:30.230 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:30.230 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:30.230 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 5686196818786205075 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:30.231 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.901248Z" 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:30.233 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.903341Z" 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:30.234 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.905919Z" 2020-10-09T17:37:30.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-10-09T17:37:30.236 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.908703Z" 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:30.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:30.238 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:30.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:30.238 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:30.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:30.238 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:30.239 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-10-09T17:37:30.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.240 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:30.281 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:30.554 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:37:30.554 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.555 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-10-09T17:37:30.555 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.901248Z" 2020-10-09T17:37:30.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.555 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.555 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.903341Z" 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.905919Z" 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.556 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.557 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-10-09T17:37:30.557 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.908703Z" 2020-10-09T17:37:30.557 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.557 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:37:30.558 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:37:30.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.559 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:30.643 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:30.644 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:30.888 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:37:30.889 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:37:30.889 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:37:30.889 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:37:30.889 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: "instance_id": 2964130218513816294 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:37:30.890 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:30.891 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:37:11.901248Z" 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.892 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:37:11.903341Z" 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:30.894 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:37:11.905919Z" 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:37:30.895 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "incremental-sync", 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:37:30.896 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:37:11.908703Z" 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:37:30.897 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 0, 2020-10-09T17:37:30.898 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 0 2020-10-09T17:37:30.898 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:30.898 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:37:30.899 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-10-09T17:37:30.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:30.900 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:30.978 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:30.979 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:31.251 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:37:31.251 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003163", 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.901248Z" 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000003959", 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.903341Z" 2020-10-09T17:37:31.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000006211", 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.905919Z" 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "00000000000000000000:00000000000000008605", 2020-10-09T17:37:31.253 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:37:11.908703Z" 2020-10-09T17:37:31.254 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:37:31.254 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:37:31.255 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T17:37:31.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:31.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:31.256 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:31.298 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:31.299 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:31.565 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:31.565 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:31.566 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:31.567 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:31.568 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:31.569 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:31.570 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:31.571 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:31.572 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:31.573 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:31.574 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:31.575 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:33.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~17s 2020-10-09T17:37:36.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:36.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:36.578 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:36.622 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:36.623 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:37.089 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:37.098 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:37.099 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:38.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~22s 2020-10-09T17:37:42.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:42.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:42.102 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:42.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:42.142 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:42.506 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:42.507 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:42.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:42.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:42.515 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:42.516 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:44.441 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~28s 2020-10-09T17:37:47.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:47.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:47.518 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:47.561 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:47.561 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:47.828 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:47.828 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:47.828 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:47.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:47.829 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:47.830 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:47.831 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:47.832 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:47.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:47.834 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:47.835 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:47.836 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:47.836 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:47.836 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:47.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:47.836 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:47.837 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:50.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~34s 2020-10-09T17:37:52.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:52.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:52.840 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:52.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:52.883 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:53.136 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:53.136 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:53.136 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:53.136 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:53.137 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:53.138 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:53.138 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:53.138 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:53.138 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:53.138 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:53.139 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:53.140 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:53.141 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.142 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:53.143 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:53.143 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:53.143 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:53.143 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:53.144 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:53.144 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:53.144 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:53.144 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:53.144 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:53.145 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:53.146 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:53.146 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:53.147 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:37:55.653 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~39s 2020-10-09T17:37:58.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:58.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:37:58.149 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:37:58.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:37:58.189 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:37:58.550 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:37:58.550 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:37:58.550 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:37:58.550 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:37:58.550 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:58.551 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:58.553 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:58.554 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:37:58.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:37:58.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:37:58.557 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:37:58.558 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:37:58.558 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:37:58.558 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:37:58.558 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:37:58.559 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:01.259 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~45s 2020-10-09T17:38:03.560 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:03.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:03.562 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:03.604 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:03.605 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:03.866 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:03.866 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:03.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:03.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:03.869 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:03.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:03.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:03.872 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:03.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:03.874 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:03.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:03.875 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:03.876 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:06.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~50s 2020-10-09T17:38:08.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:08.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:08.879 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:08.922 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:08.923 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:09.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:09.183 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:09.183 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:09.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:09.183 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:09.184 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.185 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:09.186 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:09.187 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:09.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:09.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:09.190 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:09.190 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:09.191 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:09.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:09.192 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:09.193 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:12.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~56s 2020-10-09T17:38:14.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:14.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:14.195 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:14.238 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:14.238 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:14.505 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:14.505 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:14.506 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:14.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:14.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.510 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:14.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:14.512 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:14.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:14.513 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:14.514 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:18.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~62s 2020-10-09T17:38:19.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:19.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:19.517 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:19.566 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:19.567 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:19.932 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:19.933 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:19.933 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:19.933 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:19.934 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:19.934 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:19.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:19.935 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:19.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:19.935 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:19.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:19.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:19.937 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:19.938 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:19.939 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:19.940 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:19.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:19.940 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:19.941 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:19.941 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:19.941 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:19.941 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:19.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:19.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:19.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:19.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.943 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:19.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:19.944 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:19.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:19.944 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:19.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:19.945 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:19.946 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:23.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~67s 2020-10-09T17:38:24.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:24.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:24.949 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:24.990 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:24.991 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:25.258 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:25.258 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:25.258 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:25.258 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:25.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:25.261 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:25.262 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:25.263 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:25.264 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.265 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:25.266 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:25.267 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:29.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~73s 2020-10-09T17:38:30.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:30.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:30.271 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:30.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:30.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:30.692 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:30.692 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:30.693 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:30.694 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:30.695 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:30.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:30.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:30.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:30.697 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:30.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:30.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:30.698 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:30.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.699 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:30.699 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:30.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:30.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:30.700 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:30.700 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:30.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:30.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:30.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:30.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:30.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:30.703 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:30.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:30.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:30.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:30.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.704 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.704 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:30.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:30.705 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:30.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:30.705 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:30.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:30.706 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:30.707 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:34.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~78s 2020-10-09T17:38:35.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:35.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:35.710 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:35.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:35.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:36.022 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:36.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:36.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:36.025 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:36.026 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:36.027 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:36.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:36.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:36.031 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:36.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:36.031 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:36.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:36.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:36.032 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:40.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~84s 2020-10-09T17:38:41.033 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:41.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:41.035 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:41.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:41.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:41.347 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:41.348 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:41.348 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:41.348 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:41.348 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:41.349 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:41.350 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:41.351 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:41.351 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:41.351 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:41.351 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:41.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:41.353 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:41.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:41.355 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:41.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:41.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:41.356 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:41.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:41.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:41.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:41.360 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:46.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~90s 2020-10-09T17:38:46.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:46.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:46.363 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:46.405 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:46.406 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:46.667 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:46.668 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:46.668 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:46.668 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:46.668 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:46.669 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:46.671 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:46.672 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:46.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:46.674 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:46.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:46.676 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:46.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:46.676 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:46.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:46.676 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:46.677 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:51.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:51.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:51.680 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:51.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~95s 2020-10-09T17:38:51.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:51.724 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:51.993 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:51.993 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:51.993 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:51.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:51.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:51.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:51.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:51.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:52.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:52.001 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:52.001 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:52.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:52.003 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:52.003 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:52.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:52.003 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:52.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:52.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:52.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:52.004 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:52.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:52.004 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:52.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:52.005 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:52.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:52.005 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:52.006 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:38:57.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:57.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:38:57.008 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:38:57.051 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:38:57.052 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:38:57.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~101s 2020-10-09T17:38:57.420 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:38:57.420 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:57.421 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:38:57.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:57.423 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:57.424 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:57.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:38:57.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:57.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:57.426 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:38:57.427 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:38:57.428 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:38:57.429 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:38:57.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.429 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:38:57.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:38:57.430 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:38:57.430 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:38:57.430 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:38:57.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:38:57.430 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:38:57.431 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:02.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:02.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:02.434 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:02.476 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:02.477 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:02.739 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:02.739 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:02.739 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:02.739 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:02.739 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:02.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:02.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:02.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:02.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:02.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:02.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:02.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:02.748 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:02.749 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:02.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~107s 2020-10-09T17:39:07.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:07.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:07.751 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:07.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:07.796 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:08.052 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:08.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:08.054 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:08.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:08.056 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:08.057 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:08.058 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:08.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:08.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:08.061 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:08.061 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:08.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~112s 2020-10-09T17:39:13.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:13.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:13.064 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:13.106 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:13.107 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:13.374 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:13.374 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:13.374 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:13.374 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:13.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:13.376 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:13.377 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:13.378 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:13.379 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:13.380 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.381 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:13.382 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:13.383 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:14.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~118s 2020-10-09T17:39:18.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:18.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:18.386 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:18.428 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:18.429 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:18.695 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:18.695 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:18.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:18.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:18.698 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:18.699 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:18.700 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:18.701 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:18.702 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:18.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:18.704 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:18.705 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:19.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~123s 2020-10-09T17:39:23.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:23.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:23.707 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:23.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:23.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:24.116 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:24.117 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:24.118 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:24.120 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:24.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:24.122 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:24.122 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:24.122 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:24.122 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:24.122 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:24.123 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:24.124 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.125 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:24.125 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:24.125 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:24.126 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:24.126 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:24.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:24.126 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:24.127 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:25.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~129s 2020-10-09T17:39:29.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:29.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:29.130 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:29.174 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:29.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:29.442 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:29.443 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:29.443 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:29.443 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:29.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:29.444 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:29.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:29.444 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:29.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:29.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:29.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:29.445 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:29.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:29.446 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:29.446 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:29.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:29.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:29.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.447 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:29.447 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:29.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:29.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:29.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:29.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:29.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:29.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:29.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:29.450 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:29.451 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:29.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:29.453 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.454 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:29.455 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:29.456 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:29.456 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:29.457 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:30.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~135s 2020-10-09T17:39:34.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:34.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:34.461 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:34.505 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:34.506 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:34.775 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:34.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:34.777 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:34.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:34.779 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:34.780 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.781 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:34.782 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:34.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:34.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:34.784 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:34.785 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:36.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~140s 2020-10-09T17:39:39.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:39.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:39.788 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:39.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:39.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:40.090 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:40.091 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:40.091 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:40.091 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:40.091 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:40.091 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:40.092 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:40.093 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:40.094 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:40.095 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:40.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:40.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:40.098 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:40.099 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:40.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:40.099 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:40.100 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:42.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~146s 2020-10-09T17:39:45.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:45.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:45.103 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:45.146 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:45.147 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:45.409 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:45.410 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:45.411 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:45.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:45.412 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:45.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:45.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:45.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:45.413 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:45.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:45.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:45.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:45.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:45.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:45.415 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:45.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:45.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:45.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:45.417 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:45.417 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:45.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:45.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:45.418 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:45.418 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:45.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:45.419 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:45.420 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:45.420 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:45.420 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:45.420 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:45.421 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:45.421 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:45.421 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:45.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:45.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:45.423 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:45.423 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:45.423 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:45.423 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:45.424 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:45.424 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:45.424 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:45.424 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:45.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:45.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.426 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:45.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:45.426 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:45.427 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:45.427 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:45.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:45.427 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:45.429 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:47.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~151s 2020-10-09T17:39:50.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:50.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:50.433 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:50.474 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:50.475 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:50.847 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:50.847 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:50.847 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:50.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:50.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:50.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:50.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:50.852 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:50.853 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:50.853 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:50.854 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:50.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:50.854 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:50.855 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:50.856 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:50.857 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:50.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:50.857 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:50.858 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:53.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~157s 2020-10-09T17:39:55.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:55.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:39:55.860 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:39:55.900 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:39:55.901 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:39:56.263 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:39:56.263 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:39:56.263 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:39:56.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:39:56.263 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:56.264 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.265 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:56.266 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:39:56.267 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:39:56.268 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:39:56.269 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:39:56.270 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:39:56.271 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:39:56.271 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:39:56.271 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:39:56.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:39:56.271 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:39:56.272 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:39:58.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~163s 2020-10-09T17:40:01.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:01.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:01.274 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:01.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:01.316 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:01.578 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:01.578 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:01.578 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:01.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:01.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:01.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:01.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:01.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:01.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.586 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:01.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:01.586 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:01.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:01.586 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:01.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:01.587 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:01.588 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:04.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~168s 2020-10-09T17:40:06.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:06.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:06.591 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:06.635 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:06.636 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:06.903 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:06.903 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:06.903 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:06.903 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:06.904 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:06.905 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:06.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:06.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:06.908 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:06.909 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.910 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:06.911 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:06.912 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:10.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~174s 2020-10-09T17:40:11.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:11.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:11.916 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:11.954 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:11.955 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:12.216 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:12.216 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:12.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:12.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:12.219 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:12.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:12.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:12.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.223 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:12.224 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:12.225 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:15.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~179s 2020-10-09T17:40:17.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:17.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:17.228 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:17.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:17.272 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:17.645 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:17.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:17.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:17.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:17.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:17.654 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:17.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:17.654 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:17.655 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:21.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~185s 2020-10-09T17:40:22.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:22.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:22.657 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:22.700 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:22.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:23.072 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:23.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:23.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:23.082 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:23.082 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:23.082 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:23.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:23.083 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:23.084 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:23.084 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:23.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:23.084 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:23.085 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:26.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~191s 2020-10-09T17:40:28.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:28.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:28.087 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:28.132 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:28.132 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:28.495 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:28.496 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:28.497 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:28.498 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:28.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:28.500 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:28.501 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:28.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:28.503 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:28.504 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:28.506 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:28.507 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.508 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:28.509 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:28.510 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:32.607 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~196s 2020-10-09T17:40:33.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:33.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:33.513 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:33.558 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:33.558 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:33.928 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:33.929 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:33.929 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:33.929 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:33.929 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:33.929 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:33.930 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:33.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:33.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:33.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:33.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:33.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:33.937 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:33.938 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:33.938 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:33.938 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:33.939 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:38.215 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~202s 2020-10-09T17:40:38.941 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:38.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:38.943 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:38.986 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:38.986 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:39.251 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:39.251 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:39.252 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:39.253 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:39.254 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:39.255 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:39.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:39.256 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:39.256 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:39.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:39.257 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:39.257 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:39.257 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:39.257 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:39.258 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:39.259 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:39.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:39.261 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:39.262 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:39.262 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:39.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:39.262 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:39.263 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:43.819 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~207s 2020-10-09T17:40:44.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:44.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:44.267 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:44.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:44.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:44.578 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:44.578 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:44.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:44.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:44.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:44.581 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.581 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:44.581 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:44.581 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:44.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:44.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:44.582 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:44.582 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:44.582 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:44.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:44.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:44.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:44.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:44.587 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:44.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:44.587 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:44.587 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:44.587 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:44.588 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:49.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~213s 2020-10-09T17:40:49.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:49.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:49.590 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:49.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:49.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:50.001 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:50.002 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:50.003 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:50.003 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:50.004 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:50.005 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:50.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:50.005 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:50.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:50.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:50.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:50.007 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:50.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:50.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:50.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:50.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:50.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:50.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:50.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:50.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:50.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:50.012 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:50.012 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:50.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:50.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:50.013 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:50.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:50.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:50.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:50.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:50.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:50.016 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:50.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:50.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:50.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:50.018 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:50.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:50.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:50.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:50.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:50.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:50.020 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:50.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:50.020 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:50.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:50.020 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:50.022 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:40:55.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:55.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:40:55.025 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:40:55.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~219s 2020-10-09T17:40:55.068 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:40:55.069 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:40:55.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:40:55.328 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:40:55.328 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:40:55.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:55.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:40:55.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:40:55.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:55.333 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:55.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:40:55.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:40:55.336 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:40:55.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:40:55.337 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:40:55.337 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:40:55.337 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:40:55.339 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:00.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:00.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:00.342 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:00.385 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:00.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~224s 2020-10-09T17:41:00.646 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:00.646 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:00.646 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:00.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:00.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:00.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:00.649 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:00.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:00.651 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:00.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:00.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:00.655 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:00.655 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:05.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:05.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:05.658 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:05.700 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:05.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:06.067 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:06.067 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:06.067 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:06.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:06.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:06.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:06.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:06.072 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:06.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:06.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:06.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:06.074 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.075 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:06.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:06.076 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:06.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:06.077 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:06.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:06.077 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:06.078 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:06.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~230s 2020-10-09T17:41:11.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:11.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:11.081 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:11.126 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:11.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:11.585 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:11.586 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:11.587 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:11.588 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:11.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:11.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:11.592 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.593 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:11.594 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:11.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:11.597 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:11.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:11.600 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:11.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:11.601 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:11.603 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:11.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~235s 2020-10-09T17:41:16.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:16.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:16.606 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:16.650 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:16.651 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:17.017 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:17.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:17.019 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:17.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:17.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:17.022 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:17.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:17.026 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:17.027 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:17.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~241s 2020-10-09T17:41:22.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:22.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:22.030 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:22.072 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:22.073 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:22.443 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:22.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.451 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.451 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:22.451 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:22.452 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:22.452 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:22.452 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:22.452 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:22.452 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:22.453 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:23.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~247s 2020-10-09T17:41:27.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:27.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:27.455 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:27.497 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:27.498 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:27.861 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:27.862 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:27.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:27.871 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:28.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~252s 2020-10-09T17:41:32.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:32.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:32.874 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:32.915 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:32.915 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:33.282 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:33.282 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:33.283 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:33.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:33.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:33.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:33.290 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:33.291 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:34.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~258s 2020-10-09T17:41:38.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:38.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:38.294 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:38.339 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:38.339 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:38.705 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:38.705 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:38.705 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:38.705 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:38.705 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:38.706 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:38.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:38.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:38.709 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:38.710 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:38.711 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:38.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:38.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:38.714 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:38.715 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:39.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~263s 2020-10-09T17:41:43.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:43.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:43.717 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:43.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:43.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:44.018 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:44.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:44.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:44.021 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:44.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:44.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:44.022 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:44.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.022 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:44.022 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:44.023 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:44.023 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:44.023 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:44.024 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:44.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:44.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:44.024 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:44.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:44.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:44.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:44.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:44.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:44.026 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:44.027 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:44.027 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:44.027 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:44.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:44.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:44.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:44.031 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:44.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:44.031 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:44.033 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:45.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~269s 2020-10-09T17:41:49.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:49.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:49.035 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:49.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:49.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:49.453 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:49.454 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:49.454 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:49.455 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:49.455 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:49.455 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:49.456 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:49.456 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:49.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:49.456 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:49.457 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:49.457 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:49.457 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:49.458 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:49.458 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:49.458 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:49.459 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:49.459 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:49.460 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:49.460 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:49.460 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:49.460 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:49.462 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:49.463 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:49.463 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:49.463 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:49.463 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:49.464 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:49.464 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:49.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.465 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:49.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:49.465 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:49.466 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:49.466 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:49.466 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:49.466 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:49.467 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.468 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:49.469 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:49.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:49.470 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:49.471 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:51.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~275s 2020-10-09T17:41:54.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:54.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:54.474 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:54.515 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:54.516 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:41:54.883 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:41:54.884 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:41:54.884 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:41:54.884 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:41:54.884 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:54.885 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:41:54.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:41:54.887 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:54.888 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:54.889 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:41:54.890 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.891 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:41:54.892 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:41:54.893 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:41:56.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~280s 2020-10-09T17:41:59.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:59.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:41:59.896 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:41:59.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:41:59.939 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:00.409 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:00.410 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:00.410 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:00.410 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:00.410 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:00.411 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:00.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:00.411 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:00.412 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:00.412 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:00.412 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:00.412 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:00.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:00.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:00.413 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:00.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:00.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:00.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:00.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:00.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:00.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:00.416 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:00.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:00.416 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:00.416 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:00.418 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:00.418 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:00.418 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:00.419 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:00.419 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:00.419 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:00.419 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:00.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:00.422 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:00.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:00.422 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:00.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:00.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.423 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:00.423 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:00.423 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:00.424 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:00.424 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:00.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:00.424 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:00.426 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:02.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~286s 2020-10-09T17:42:05.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:05.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:05.428 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:05.471 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:05.472 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:05.843 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:05.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:05.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:05.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:05.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:05.848 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:05.850 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:05.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:05.852 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:05.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:05.852 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:05.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:05.852 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:05.853 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:07.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~291s 2020-10-09T17:42:10.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:10.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:10.856 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:10.899 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:10.900 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:11.257 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:11.259 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:11.259 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:11.259 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:11.259 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:11.259 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:11.264 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.265 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.266 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:11.266 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:11.266 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:11.266 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:11.266 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:11.267 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:11.267 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:11.268 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:13.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~297s 2020-10-09T17:42:16.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:16.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:16.270 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:16.684 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:16.684 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:16.685 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:16.686 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:16.687 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:16.688 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:16.689 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:16.690 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:16.691 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:16.692 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:16.693 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:16.694 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:19.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~303s 2020-10-09T17:42:19.122 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2020-10-09T17:42:19.122 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/daemonwatchdog.py", line 37, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/daemonwatchdog.py", line 113, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/daemonwatchdog.py", line 53, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 254, in __getattr__ raise AttributeError(name) AttributeError: mounts 2020-10-09T17:42:21.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:21.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:21.696 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:21.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:21.740 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:22.009 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:22.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:22.010 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:22.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:22.010 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:22.011 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:22.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:22.014 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:22.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:22.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:22.018 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:22.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:22.018 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:22.019 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:27.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:27.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:27.023 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:27.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:27.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:27.428 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:27.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:27.430 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:27.431 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:27.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:27.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:27.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:27.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:27.436 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:27.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:27.437 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:27.438 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:32.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:32.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:32.441 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:32.483 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:32.484 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:32.848 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:32.850 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:32.851 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:32.852 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:32.853 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:32.854 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:32.854 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:32.854 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:32.855 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:32.856 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.857 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:32.858 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:32.859 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:37.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:37.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:37.862 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:37.897 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:37.898 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:38.158 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:38.158 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:38.159 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:38.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:38.167 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:38.169 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:43.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:43.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:43.174 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:43.216 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:43.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:43.481 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:43.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:43.483 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:43.485 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.486 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:43.487 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:43.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:43.489 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:43.490 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:43.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:43.491 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:43.492 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:48.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:48.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:48.494 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:48.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:48.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:49.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:49.003 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:49.003 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:49.003 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:49.003 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:49.003 INFO:teuthology.orchestra.run.smithi184.stdout: "instance_id": 9637575806026136719 2020-10-09T17:42:49.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:49.004 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:49.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:49.004 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:49.004 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000000780", 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:27:07.142967Z" 2020-10-09T17:42:49.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:49.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:49.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:49.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:49.006 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000002374", 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:22:13.770773Z" 2020-10-09T17:42:49.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:49.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:49.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:49.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:49.008 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:49.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000003835", 2020-10-09T17:42:49.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:49.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:49.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:49.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:29:36.683330Z" 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "incremental-sync", 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "00000000000000000000:00000000000000006229", 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:32:50.910879Z" 2020-10-09T17:42:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 0, 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 0 2020-10-09T17:42:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:49.013 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:49.014 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=00000000000000000000:00000000000000003163 target=00000000000000000000:00000000000000000780, shard=1 master=00000000000000000000:00000000000000003959 target=00000000000000000000:00000000000000002374, shard=2 master=00000000000000000000:00000000000000006211 target=00000000000000000000:00000000000000003835, shard=3 master=00000000000000000000:00000000000000008605 target=00000000000000000000:00000000000000006229 2020-10-09T17:42:54.016 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... FAIL 2020-10-09T17:42:54.017 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-59 2020-10-09T17:42:54.061 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-60 2020-10-09T17:42:54.139 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-61 2020-10-09T17:42:54.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:54.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:54.788 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:54.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:55.041 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:42:55.065 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:42:55.066 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.066 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:42:55.066 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:42:55.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.067 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:42:55.067 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:42:55.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.068 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.068 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:42:55.068 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:42:55.068 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.069 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.069 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602265374.173440_353.1", 2020-10-09T17:42:55.069 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:42:54.173440Z" 2020-10-09T17:42:55.069 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.070 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:42:55.071 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:42:55.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.073 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:55.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:55.127 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:55.363 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:55.363 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:55.363 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:55.364 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:55.364 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:55.364 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:42:55.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:42:55.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:42:55.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:55.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:55.367 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:55.368 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:55.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265374.173440_353.1", 2020-10-09T17:42:55.370 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:42:54.173440Z", 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.371 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:55.372 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:55.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:42:55.376 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:42:55.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.377 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:55.420 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:55.420 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:55.681 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:42:55.682 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:42:55.682 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:42:55.683 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:42:55.683 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:42:55.683 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:42:55.683 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:42:55.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.684 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:42:55.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.684 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:42:55.684 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:42:55.685 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:42:55.685 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:42:55.685 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:42:55.685 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:42:55.686 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:42:55.686 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:42:55.686 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:42:55.686 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.687 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:42:55.687 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:42:55.688 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:42:55.688 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:42:55.688 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:42:55.688 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:42:55.689 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:42:55.689 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:42:55.689 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:42:55.690 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.690 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:42:55.690 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:42:55.691 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:42:55.691 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:42:55.691 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:42:55.691 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:42:55.692 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:42:55.692 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:42:55.692 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:42:55.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:42:55.693 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:42:55.693 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:42:55.694 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:42:55.694 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602265374.173440_353.1", 2020-10-09T17:42:55.694 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:42:55.695 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:42:55.695 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:42:55.695 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:42:54.173440Z", 2020-10-09T17:42:55.695 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:42:55.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.696 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.696 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:42:55.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:42:55.697 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:42:55.697 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:42:55.697 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:42:55.697 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:42:55.698 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:42:55.699 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:42:55.700 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:42:55.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:42:55.704 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:42:55.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:42:55.750 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:42:56.114 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:42:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:42:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:42:56.114 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:42:56.115 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:56.116 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:42:56.117 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:42:56.118 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:56.119 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:56.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:42:56.121 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:42:56.122 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:42:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:42:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:42:56.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:42:56.123 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:42:56.125 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:01.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:01.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:01.128 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:01.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:01.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:01.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:01.546 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:01.547 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:06.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:06.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:06.550 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:06.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:06.590 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:06.952 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:06.952 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:06.953 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:06.954 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:06.955 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:06.956 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:06.957 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:06.957 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:06.957 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:06.958 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:06.959 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:06.960 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:06.961 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:06.961 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:06.961 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.961 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:06.962 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:06.962 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:06.962 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:06.963 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:06.963 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:06.963 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:06.963 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:06.964 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:06.964 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:06.964 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:06.964 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.965 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:06.965 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:06.965 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:06.970 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:06.971 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:11.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:11.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:11.974 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:12.015 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:12.279 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:12.279 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:12.280 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:12.281 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:12.282 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:12.283 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:12.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:12.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:12.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:12.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:12.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:12.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:12.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:12.292 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:12.292 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:17.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:17.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:17.295 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:17.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:17.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:17.700 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:17.700 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:17.701 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:17.702 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:17.703 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:17.704 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:17.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:17.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:17.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.708 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:17.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:17.710 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:22.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:22.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:22.714 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:22.757 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:22.757 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:23.012 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:23.012 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:23.013 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:23.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:23.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:23.016 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:23.017 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:23.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:23.019 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:23.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:23.021 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:23.022 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:23.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:23.023 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:28.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:28.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:28.026 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:28.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:28.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:28.325 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:28.326 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:28.327 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:28.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:28.327 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:28.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:28.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:28.328 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:28.328 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:28.328 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:28.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:28.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:28.329 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:28.329 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:28.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:28.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:28.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:28.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:28.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:28.331 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:28.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:28.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:28.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:28.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:28.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:28.334 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:28.335 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:28.336 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:28.337 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:28.338 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:28.339 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:28.340 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:28.341 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:33.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:33.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:33.343 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:33.383 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:33.383 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:33.753 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:33.753 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:33.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:33.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:33.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:33.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:33.758 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:33.759 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:33.760 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.761 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:33.762 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:33.763 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:38.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:38.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:38.766 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:38.805 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:38.806 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:39.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:39.166 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:39.166 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:39.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:39.166 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:39.166 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:39.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:39.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:39.169 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:39.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:39.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:39.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:39.173 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:39.174 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:39.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:39.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:39.176 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:44.177 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:44.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:44.179 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:44.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:44.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:44.475 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:44.475 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:44.475 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:44.475 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:44.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:44.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:44.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:44.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:44.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:44.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:44.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:44.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:44.484 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:44.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:44.484 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:44.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:44.484 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:44.485 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:49.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:49.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:49.488 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:49.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:49.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:49.796 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:49.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:49.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:49.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:49.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:49.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:49.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:49.803 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.804 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:49.805 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:49.806 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:43:54.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:54.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:43:54.809 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:43:54.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:43:54.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:43:55.208 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:43:55.208 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:55.209 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:43:55.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:55.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:43:55.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:55.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:55.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:55.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:55.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:55.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:43:55.214 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:55.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:43:55.216 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:43:55.217 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:43:55.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:43:55.218 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:43:55.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:43:55.218 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:43:55.219 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:00.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:00.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:00.222 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:00.261 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:00.261 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:00.633 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:00.634 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:00.635 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:00.636 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:00.636 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:00.636 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:00.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:00.636 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:00.637 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:00.638 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:00.639 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:00.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:00.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:00.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:00.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:00.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:00.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:05.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:05.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:05.647 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:05.687 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:05.687 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:06.054 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:06.054 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:06.055 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:06.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:06.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:06.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:06.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:06.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:06.061 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:06.062 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:06.063 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:06.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:06.064 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:06.065 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:11.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:11.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:11.067 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:11.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:11.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:11.368 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:11.368 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:11.368 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:11.368 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:11.369 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:11.370 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:11.371 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:11.372 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:11.373 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:11.374 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:11.375 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:11.376 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:11.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:11.377 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:11.378 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:16.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:16.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:16.380 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:16.420 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:16.420 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:17.024 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:17.024 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:17.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:17.026 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:17.027 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:17.028 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:17.029 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:17.030 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:17.031 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.032 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:17.033 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:17.034 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:22.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:22.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:22.037 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:22.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:22.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:22.441 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:22.441 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:22.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:22.443 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:22.444 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:22.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:22.446 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:22.447 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:22.448 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:22.449 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:22.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:22.450 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:22.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:27.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:27.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:27.454 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:27.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:27.495 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:27.863 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:27.863 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:27.864 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:27.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:27.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:27.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:27.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:27.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:27.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:27.872 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:27.873 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:32.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:32.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:32.875 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:32.916 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:32.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:33.287 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:33.287 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:33.288 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:33.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:33.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:33.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:33.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:33.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:33.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.295 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:33.296 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:33.297 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:38.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:38.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:38.300 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:38.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:38.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:38.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:38.604 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:38.604 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:38.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:38.604 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:38.605 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:38.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:38.607 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:38.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:38.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:38.610 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:38.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:38.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:38.613 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:38.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:43.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:43.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:43.616 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:43.658 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:43.658 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:43.925 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:43.926 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:43.926 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:43.926 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:43.926 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:43.926 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:43.927 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:43.928 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:43.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:43.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:43.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:43.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:43.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:43.935 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:43.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:43.935 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:43.936 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:48.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:48.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:48.939 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:48.978 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:48.979 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:49.344 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:49.344 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:49.345 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:49.345 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:49.345 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:49.345 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:49.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:49.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:49.346 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:49.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:49.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:49.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:49.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:49.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:49.349 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:49.349 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:49.350 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:49.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:49.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:49.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:49.355 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:49.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:49.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:49.356 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:49.356 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:49.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.357 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:49.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:49.357 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:49.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:49.358 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:49.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:49.358 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:49.359 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:54.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:54.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:54.362 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:54.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:54.402 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:54.665 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:54.666 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:54.666 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:54.666 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:54.666 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:54.667 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:54.667 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:54.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:54.667 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:54.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:54.668 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:54.668 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:54.668 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:54.668 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:54.669 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:54.670 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:54.670 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:54.670 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:54.670 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:54.671 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:54.672 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:54.673 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:54.673 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:54.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:54.673 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:54.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:54.675 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:54.676 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:44:54.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:44:54.677 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:44:54.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:54.677 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:44:54.678 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:44:59.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:59.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:44:59.680 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:44:59.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:44:59.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:44:59.990 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:44:59.991 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:44:59.991 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:44:59.991 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:44:59.991 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:44:59.992 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:44:59.992 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:59.992 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:59.992 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:44:59.992 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:44:59.993 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:44:59.994 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:44:59.995 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:59.996 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:59.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:44:59.998 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:44:59.999 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:00.000 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:00.000 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:00.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:00.000 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:00.001 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:05.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:05.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:05.004 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:05.047 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:05.048 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:05.401 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:05.402 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:05.403 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:05.404 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:05.405 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:05.406 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:05.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:05.408 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.409 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:05.410 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:05.411 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:10.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:10.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:10.414 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:10.481 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:10.481 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:10.748 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:10.749 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:10.750 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:10.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:10.752 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:10.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:10.754 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:10.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:10.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:10.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:10.758 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:10.759 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:15.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:15.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:15.762 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:15.802 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:15.803 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:16.061 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:16.061 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:16.062 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:16.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:16.064 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:16.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:16.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:16.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:16.069 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:16.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:16.070 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:16.071 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:21.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:21.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:21.073 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:21.113 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:21.113 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:21.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:21.471 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:21.471 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:21.471 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:21.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:21.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:21.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:21.475 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:21.476 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:21.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:21.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:21.479 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:21.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:21.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:21.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:21.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:21.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:21.480 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:21.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.480 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:21.481 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:21.482 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:26.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:26.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:26.485 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:26.525 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:26.525 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:26.895 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:26.895 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:26.895 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:26.895 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:26.895 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:26.896 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:26.897 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:26.898 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:26.900 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:26.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:26.902 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.903 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:26.904 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:26.905 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:31.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:31.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:31.908 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:31.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:31.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:32.215 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:32.216 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:32.216 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:32.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:32.216 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:32.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:32.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:32.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:32.220 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:32.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:32.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:32.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:32.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:32.225 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:32.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:32.225 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:32.226 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:37.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:37.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:37.229 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:37.270 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:37.271 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:37.535 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:37.536 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:37.537 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.538 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:37.539 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:37.540 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:37.541 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:37.542 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.543 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:37.544 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:37.545 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:42.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:42.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:42.548 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:42.588 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:42.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:42.962 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:42.962 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:42.963 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:42.963 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:42.963 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:42.963 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:42.963 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:42.964 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:42.965 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:42.966 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:42.968 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:42.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:42.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:42.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:42.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.971 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:42.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:42.972 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:42.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:42.972 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:42.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:42.972 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:42.973 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:47.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:47.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:47.976 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:48.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:48.016 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:48.277 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:48.277 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:48.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:48.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:48.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:48.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:48.287 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:48.288 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:53.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:53.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:53.291 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:53.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:53.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:53.596 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:53.596 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:53.596 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:53.596 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:53.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:53.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:53.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:53.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:53.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:53.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:53.600 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:53.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:53.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:53.603 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:53.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:53.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:53.606 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:53.607 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:45:58.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:58.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:45:58.609 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:45:58.651 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:45:58.651 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:45:59.015 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:45:59.015 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:45:59.015 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:45:59.015 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:59.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:59.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:59.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:59.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:45:59.020 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:45:59.021 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:45:59.022 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:45:59.023 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:45:59.024 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:45:59.024 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:45:59.024 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:45:59.025 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:04.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:04.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:04.028 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:04.071 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:04.073 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:04.431 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:04.431 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:04.432 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:04.432 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:04.432 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:04.432 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:04.432 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:04.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:04.433 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:04.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:04.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:04.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:04.434 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:04.434 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:04.434 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:04.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:04.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:04.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:04.436 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:04.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:04.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:04.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:04.437 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:04.437 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:04.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:04.438 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:04.438 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:04.438 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:04.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:04.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:04.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:04.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:04.440 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:04.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:04.441 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:04.441 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:04.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:04.441 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:04.441 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:04.442 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:04.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:04.444 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:04.445 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:04.445 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:04.445 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:04.446 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:09.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:09.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:09.449 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:09.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:09.490 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:09.749 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:09.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:09.751 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:09.753 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:09.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:09.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:09.756 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:09.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:09.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:09.759 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:09.760 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:14.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:14.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:14.762 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:14.804 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:14.805 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:15.172 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:15.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:15.174 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:15.175 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:15.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:15.177 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:15.178 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:15.179 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.180 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:15.181 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:15.183 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:20.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:20.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:20.185 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:20.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:20.226 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:20.590 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:20.591 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:20.592 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:20.593 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:20.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:20.595 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:20.596 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:20.597 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:20.598 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:20.598 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:20.598 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:20.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:20.599 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:20.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:20.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.601 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.601 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:20.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:20.602 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:20.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:20.602 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:20.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:20.602 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:20.604 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:25.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:25.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:25.607 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:25.649 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:25.650 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:26.011 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:26.011 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:26.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:26.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:26.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:26.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:26.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:26.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:26.018 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.019 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:26.020 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:26.021 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:31.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:31.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:31.023 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:31.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:31.066 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:31.328 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:31.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:31.333 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:31.334 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:31.335 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:31.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:31.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:31.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:31.479 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:31.480 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:36.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:36.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:36.483 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:36.525 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:36.791 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:36.792 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:36.792 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:36.792 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:36.792 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:36.792 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:36.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:36.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:36.793 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:36.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:36.793 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:36.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:36.795 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:36.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:36.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:36.798 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:36.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:36.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:36.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:36.802 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:36.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:36.802 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:36.803 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:41.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:41.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:41.805 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:41.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:41.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:42.219 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:42.219 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:42.219 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:42.219 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:42.219 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:42.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:42.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:42.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:42.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:42.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:42.228 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:42.229 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:47.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:47.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:47.231 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:47.273 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:47.274 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:47.647 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:47.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:47.654 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:47.655 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:47.656 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:47.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:47.657 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:47.658 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:52.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:52.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:52.662 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:52.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:52.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:52.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:52.967 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:52.968 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:52.969 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:52.970 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:52.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:52.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:52.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:52.974 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.975 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:52.976 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:52.977 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:46:57.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:57.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:46:57.980 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:46:58.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:46:58.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:46:58.284 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:58.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:58.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:58.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:58.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:46:58.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:46:58.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:46:58.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.292 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:46:58.293 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:46:58.294 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:03.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:03.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:03.297 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:03.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:03.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:03.605 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:03.606 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:03.607 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:03.607 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:03.607 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:03.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:03.608 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:03.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:03.608 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:03.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:03.609 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:03.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:03.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:03.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:03.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:03.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:03.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:03.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:03.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:03.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:03.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:03.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:03.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:03.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:03.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:03.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:03.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:03.616 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:03.616 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:03.616 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:03.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.617 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:03.617 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:03.617 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:03.618 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:03.618 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:03.618 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:03.619 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:03.619 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:03.619 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:03.619 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:03.620 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:03.621 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:03.621 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:03.621 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:03.621 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:03.622 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:08.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:08.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:08.624 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:08.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:08.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:09.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:09.031 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:09.031 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:09.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:09.031 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:09.031 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:09.032 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:09.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:09.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:09.035 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:09.035 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:09.035 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:09.035 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:09.035 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:09.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:09.037 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:09.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:09.039 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:09.040 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:09.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:09.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:09.042 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:14.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:14.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:14.045 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:14.086 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:14.086 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:14.351 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:14.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:14.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:14.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:14.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:14.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:14.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:14.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:14.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:14.360 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:14.360 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:14.361 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:19.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:19.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:19.366 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:19.408 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:19.409 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:19.777 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:19.777 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:19.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:19.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:19.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:19.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:19.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:19.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.785 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:19.786 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:19.787 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:24.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:24.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:24.791 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:24.831 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:24.832 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:25.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:25.093 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:25.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:25.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:25.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:25.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:25.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:25.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:25.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:25.102 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:25.103 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:25.103 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:25.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:25.104 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:25.104 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:25.104 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:25.105 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:25.106 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.107 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:25.108 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:25.109 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:25.110 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:30.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:30.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:30.112 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:30.156 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:30.156 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:30.525 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:30.525 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:30.525 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:30.525 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:30.526 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:30.527 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:30.528 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.529 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:30.530 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:30.530 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:30.530 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:30.530 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:30.530 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.531 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:30.532 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:30.532 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:30.532 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:30.532 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:30.532 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.533 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:30.534 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:30.535 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:30.535 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:30.535 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:30.536 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:35.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:35.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:35.539 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:35.580 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:35.580 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:35.845 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:35.845 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:35.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:35.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:35.849 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:35.850 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:35.851 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:35.852 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:35.853 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:35.854 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:35.855 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:35.855 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:35.855 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:35.855 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:35.856 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:40.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:40.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:40.859 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:40.901 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:40.902 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:41.271 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:41.271 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:41.272 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:41.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:41.274 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:41.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:41.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:41.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:41.274 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:41.275 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:41.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:41.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:41.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:41.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:41.280 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:41.281 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:46.283 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:46.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:46.285 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:46.325 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:46.325 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:46.588 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:46.589 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:46.590 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:46.591 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:46.592 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:46.592 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:46.592 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:46.592 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.592 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:46.593 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:46.594 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:46.595 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:46.595 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:46.595 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:46.595 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:46.595 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:46.596 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:46.597 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:46.597 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:46.597 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:46.597 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:46.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:46.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:46.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:46.598 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:46.599 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:46.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:46.600 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:46.601 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:51.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:51.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:51.603 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:51.646 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:51.646 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:51.911 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:51.911 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:51.911 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:51.911 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:51.912 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:51.913 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:51.914 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:51.915 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:51.916 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:51.917 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:51.918 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.919 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:51.920 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:51.921 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:47:56.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:56.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:47:56.924 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:47:56.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:47:56.964 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:47:57.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:47:57.230 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:47:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:47:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:47:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:47:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:47:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:47:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:47:57.239 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:47:57.240 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:02.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:02.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:02.243 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:02.284 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:02.284 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:02.649 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:02.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:02.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:02.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:02.654 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:02.654 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:02.654 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:02.654 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:02.655 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:02.656 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:02.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:02.658 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:02.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:02.660 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:02.661 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:07.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:07.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:07.664 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:07.706 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:07.706 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:07.969 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:07.969 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:07.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:07.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:07.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:07.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:07.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:07.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:07.976 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:07.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:07.979 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:07.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:12.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:12.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:12.982 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:13.037 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:13.037 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:13.408 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:13.408 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:13.408 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:13.408 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:13.409 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:13.410 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:13.411 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:13.412 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:13.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:13.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:13.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:13.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:13.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:13.417 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:13.418 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265374.173440_353.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:18.420 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... FAIL 2020-10-09T17:48:18.421 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-62 2020-10-09T17:48:18.464 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-63 2020-10-09T17:48:18.534 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-64 2020-10-09T17:48:18.627 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:48:18.622+0000 7f0c93ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:48:18.628 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:48:18.626+0000 7fd91dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:48:19.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.176 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:19.219 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:19.220 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:19.421 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:48:19.454 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:19.455 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:19.456 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:19.457 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602265698.570765_374.1", 2020-10-09T17:48:19.457 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:48:18.570765Z" 2020-10-09T17:48:19.457 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:19.457 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:48:19.458 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:48:19.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.459 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:19.500 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:19.500 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:19.749 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:19.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:19.751 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:19.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:19.753 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:19.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:19.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:19.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265698.570765_374.1", 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:19.757 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:48:18.570765Z", 2020-10-09T17:48:19.758 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:19.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.759 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:19.759 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:19.759 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:19.759 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:19.760 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:19.760 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:19.762 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:48:19.762 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:48:19.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:19.763 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:19.797 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:19.797 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:20.048 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:20.049 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:48:20.050 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:48:20.052 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:48:20.053 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:20.054 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:48:20.055 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602265698.570765_374.1", 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:48:18.570765Z", 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.056 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:48:20.057 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:48:20.058 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:48:20.059 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:48:20.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:20.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:20.060 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:20.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:20.102 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:20.469 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:20.470 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:20.471 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:20.472 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:20.472 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:20.472 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:20.472 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:20.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:20.473 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:20.473 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:20.474 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:20.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:20.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:20.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:20.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:20.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:20.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:20.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.478 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:20.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:20.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:20.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:20.479 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:20.479 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:20.479 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:20.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:20.480 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:20.480 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:20.480 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:20.480 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.480 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:20.481 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:20.481 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:20.481 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:20.481 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:20.482 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:20.482 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:20.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:20.482 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:20.483 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:20.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:20.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.483 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.484 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:20.484 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:20.484 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:20.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:20.484 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:20.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:20.485 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:20.486 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:25.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:25.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:25.490 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:25.531 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:25.532 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:25.901 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:25.902 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:25.903 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:25.904 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:25.905 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:25.906 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:25.907 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:25.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:25.909 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.910 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:25.911 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:25.912 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:30.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:30.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:30.915 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:30.953 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:30.953 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:31.321 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:31.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:31.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:31.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:31.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:31.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:31.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:31.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:31.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:31.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:31.332 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:31.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:36.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:36.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:36.336 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:36.377 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:36.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:36.749 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:36.749 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:36.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:36.751 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:36.752 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:36.754 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:36.755 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:36.756 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:36.757 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:36.758 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:36.758 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:36.759 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:41.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:41.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:41.761 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:41.801 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:41.801 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:42.161 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:42.226 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:42.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:42.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.229 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:42.230 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:42.231 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:42.232 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:42.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:42.234 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:42.235 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:42.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:42.236 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:42.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:47.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:47.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:47.239 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:47.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:47.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:47.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:47.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:47.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:47.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:47.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:47.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:47.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:47.650 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:47.652 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:47.653 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:52.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:52.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:52.657 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:52.697 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:53.060 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:53.060 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:53.061 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:53.062 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:53.063 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:53.064 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:53.064 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:53.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:53.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:53.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:53.069 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:53.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:53.070 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:53.071 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:48:58.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:58.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:48:58.075 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:48:58.119 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:48:58.120 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:48:58.386 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:48:58.386 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:58.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:48:58.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:58.389 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:58.390 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:58.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:58.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:48:58.393 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:48:58.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:48:58.395 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:48:58.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:48:58.396 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:48:58.397 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:03.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:03.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:03.399 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:03.439 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:03.440 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:03.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:03.709 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:03.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:03.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:03.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:03.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:03.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:03.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:03.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:03.717 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:03.718 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:03.718 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:03.719 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:08.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:08.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:08.721 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:08.763 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:08.763 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:09.132 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:09.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:09.133 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:09.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:09.135 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:09.137 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:09.138 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:09.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:09.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:09.141 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:09.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:09.141 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:09.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:14.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:14.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:14.145 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:14.186 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:14.187 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:14.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:14.621 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:14.622 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.623 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:14.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:14.625 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:14.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:14.628 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:14.629 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:14.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:14.630 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:14.631 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:19.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:19.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:19.634 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:19.676 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:19.676 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:20.039 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:20.039 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:20.040 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:20.041 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:20.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:20.043 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:20.044 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:20.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:20.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:20.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:20.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:20.048 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:20.049 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:25.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:25.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:25.052 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:25.092 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:25.093 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:25.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:25.362 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:25.362 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:25.362 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:25.362 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:25.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:25.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:25.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:25.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:25.367 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:25.368 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:25.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:25.370 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:25.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:25.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:25.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:30.373 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:30.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:30.374 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:30.413 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:30.414 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:30.785 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:30.785 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:30.785 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:30.786 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:30.787 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:30.788 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:30.789 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:30.790 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.791 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:30.792 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:30.793 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:30.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:30.794 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:30.795 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:35.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:35.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:35.797 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:35.836 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:35.837 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:36.203 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:36.204 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:36.205 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:36.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:36.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:36.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:36.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:36.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.211 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:36.212 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:36.213 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:41.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:41.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:41.216 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:41.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:41.259 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:41.623 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:41.623 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:41.623 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:41.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:41.625 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:41.626 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:41.626 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:41.626 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:41.626 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:41.627 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:41.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:41.630 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:41.631 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:41.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:41.633 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:41.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:41.633 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:41.634 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:46.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:46.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:46.637 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:46.677 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:46.677 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:46.936 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:46.937 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:46.937 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:46.937 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:46.937 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:46.937 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:46.938 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:46.939 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:46.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:46.941 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:46.942 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:46.943 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.944 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:46.945 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:46.946 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:51.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:51.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:51.950 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:51.992 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:51.993 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:52.359 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:52.359 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:52.359 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:52.360 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:52.360 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:52.360 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:52.360 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:52.361 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:52.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:52.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:52.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:52.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:52.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:52.365 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.365 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:52.365 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:52.365 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:52.366 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:52.367 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:52.368 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:52.369 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:52.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:52.371 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:52.372 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:49:57.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:57.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:49:57.375 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:49:57.416 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:49:57.416 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:49:57.788 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:49:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:49:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:49:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:49:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:49:57.789 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:49:57.790 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:49:57.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:49:57.792 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:57.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:49:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:49:57.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:49:57.796 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:49:57.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:57.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:49:57.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:49:57.798 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:49:57.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:49:57.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:49:57.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:49:57.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:49:57.799 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:49:57.799 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:49:57.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:49:57.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.800 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:49:57.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:49:57.801 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:49:57.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:49:57.801 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:49:57.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:49:57.801 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:49:57.803 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:02.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:02.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:02.805 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:02.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:03.194 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:03.195 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:03.195 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:03.195 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:03.195 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:03.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:03.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:03.198 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.199 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:03.200 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:03.201 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:03.202 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:03.203 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:03.204 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:03.204 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:03.205 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:08.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:08.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:08.208 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:08.251 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:08.251 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:08.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:08.621 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:08.621 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:08.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:08.622 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:08.622 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:08.622 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:08.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:08.623 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:08.623 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:08.623 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:08.623 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:08.624 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:08.625 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:08.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:08.626 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:08.626 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:08.626 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:08.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:08.627 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:08.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:08.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:08.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:08.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:08.629 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:08.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:08.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:08.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:08.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:08.630 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:08.630 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:08.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:08.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:08.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:08.632 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:08.633 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:08.633 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:08.633 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:08.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:08.633 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:08.634 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:13.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:13.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:13.638 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:13.679 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:13.680 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:14.043 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:14.044 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:14.045 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:14.046 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:14.047 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:14.048 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:14.049 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.050 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:14.051 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.052 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:14.053 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:14.054 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:19.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:19.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:19.057 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:19.097 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:19.098 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:19.457 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:19.458 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:19.458 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:19.458 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:19.458 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:19.459 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:19.460 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:19.461 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:19.462 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:19.462 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:19.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:19.463 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:19.463 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:19.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:19.464 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:19.464 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:19.465 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:19.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:19.465 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:19.465 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:19.466 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:19.466 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:19.466 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:19.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:19.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:19.468 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:19.469 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:19.469 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:19.470 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:19.470 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:19.471 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:19.471 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:19.472 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:19.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:19.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:19.473 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:19.475 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:24.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:24.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:24.477 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:24.516 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:24.517 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:24.881 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:24.881 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:24.882 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:24.883 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:24.884 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:24.885 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:24.886 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:24.887 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:24.888 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.889 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:24.890 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:24.891 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:29.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:29.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:29.893 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:29.933 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:29.934 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:30.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:30.203 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:30.204 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:30.204 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:30.205 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:30.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:30.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:30.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:30.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:30.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:30.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:30.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:30.208 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:30.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:30.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:30.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:30.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:30.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:30.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:30.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:30.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:30.211 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:30.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:30.212 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:30.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:30.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:30.213 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:30.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:30.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:30.214 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:30.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:30.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:30.215 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:30.215 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:30.216 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:30.216 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:30.216 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:30.216 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:30.216 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:30.217 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:30.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:30.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.218 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.218 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:30.218 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:30.218 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:30.219 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:30.219 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:30.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:30.220 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:30.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:35.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:35.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:35.224 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:35.264 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:35.264 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:35.634 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:35.635 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:35.635 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:35.635 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:35.635 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:35.636 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:35.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:35.638 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:35.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:35.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:35.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:35.643 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:35.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:35.644 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:35.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:35.644 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:35.645 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:40.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:40.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:40.648 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:40.701 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:40.702 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:40.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:40.967 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:40.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:40.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:40.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:40.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:40.970 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:40.971 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:40.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:40.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:40.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:40.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:40.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:40.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:40.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:40.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:40.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:40.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:40.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:40.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:40.976 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:40.976 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:40.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:40.976 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:40.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:40.977 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:40.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:40.978 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:40.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:40.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:40.979 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:40.979 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:40.979 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:40.979 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:40.980 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:40.980 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:40.980 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:40.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:40.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:40.982 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:40.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:45.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:45.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:45.985 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:46.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:46.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:46.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:46.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:46.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:46.293 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:46.294 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:46.295 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:46.296 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:46.297 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:46.298 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:46.299 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:46.300 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:46.300 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:46.300 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:46.301 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:51.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:51.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:51.303 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:51.344 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:51.345 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:51.809 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:51.810 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:51.811 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:51.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:51.813 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:51.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:51.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:51.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:51.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:51.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:51.816 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:51.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:51.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:51.819 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:51.820 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:50:56.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:56.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:50:56.822 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:50:56.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:50:56.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:50:57.230 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:50:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:50:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:50:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:50:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:50:57.231 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:50:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:50:57.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:50:57.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:57.234 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:57.235 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:57.236 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:50:57.237 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:50:57.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:50:57.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:50:57.240 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:50:57.240 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:50:57.240 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:50:57.240 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:50:57.240 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:50:57.241 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:02.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:02.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:02.244 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:02.279 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:02.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:02.641 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:02.641 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:02.641 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:02.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:02.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:02.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:02.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:02.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:02.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:02.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:02.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.649 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:02.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:02.651 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:07.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:07.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:07.654 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:07.721 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:07.722 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:07.994 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:07.995 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:07.995 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:07.995 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:07.996 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:07.996 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:07.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:07.997 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:07.997 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:07.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:07.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:07.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:07.999 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:07.999 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:07.999 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:07.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:08.000 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:08.000 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:08.000 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:08.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:08.001 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:08.001 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:08.001 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:08.001 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:08.002 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:08.003 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:08.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:08.005 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:08.005 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:08.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:08.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:08.006 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:08.006 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:08.006 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:08.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:08.008 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:08.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:08.009 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:08.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:13.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:13.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:13.014 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:13.053 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:13.053 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:13.419 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:13.420 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:13.421 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:13.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:13.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:13.424 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:13.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:13.426 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:13.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:13.428 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:13.428 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:13.428 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:13.428 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:13.428 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:13.429 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:18.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:18.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:18.432 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:18.472 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:18.472 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:18.736 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:18.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:18.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:18.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:18.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:18.742 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:18.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:18.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:18.745 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:18.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:18.746 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:18.747 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:23.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:23.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:23.749 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:23.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:23.790 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:24.154 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:24.155 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:24.155 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:24.155 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:24.155 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:24.155 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:24.156 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:24.157 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:24.158 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:24.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:24.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:24.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:24.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:24.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:24.164 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:24.164 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:29.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:29.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:29.167 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:29.209 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:29.210 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:29.575 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:29.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:29.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:29.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:29.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:29.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:29.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:29.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:29.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:29.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:29.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:29.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:29.584 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:29.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:29.585 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:29.586 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:34.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:34.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:34.588 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:34.631 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:34.631 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:35.000 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:35.000 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:35.001 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:35.002 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:35.003 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:35.004 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:35.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:35.006 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:35.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:35.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.009 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:35.010 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:35.011 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:40.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:40.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:40.014 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:40.056 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:40.057 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:40.427 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:40.427 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:40.427 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:40.427 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:40.428 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:40.429 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:40.429 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:40.429 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:40.429 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:40.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:40.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:40.432 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:40.433 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:40.434 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.435 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:40.436 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:40.437 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:45.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:45.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:45.440 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:45.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:45.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:45.740 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:45.740 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:45.741 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:45.742 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:45.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:45.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:45.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:45.746 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:45.747 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:45.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:45.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.749 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:45.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:45.749 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:45.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:45.750 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:45.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:45.750 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:45.751 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:50.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:50.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:50.753 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:50.794 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:50.795 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:51.157 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:51.158 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:51.158 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:51.158 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:51.158 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:51.159 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:51.160 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:51.161 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:51.162 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:51.163 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:51.164 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:51.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:51.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:51.167 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:51.167 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:51:56.168 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:56.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:51:56.170 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:51:56.208 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:51:56.208 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:51:56.572 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:56.573 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:56.574 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:56.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:51:56.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:56.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:51:56.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:51:56.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:51:56.582 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:51:56.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:51:56.583 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:51:56.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:01.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:01.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:01.586 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:01.632 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:01.633 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:02.000 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:02.001 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:02.001 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:02.001 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:02.002 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:02.002 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:02.002 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:02.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:02.003 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:02.003 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:02.003 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:02.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:02.004 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:02.004 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:02.004 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:02.004 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:02.004 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:02.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:02.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:02.005 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:02.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:02.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:02.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:02.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:02.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:02.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:02.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:02.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:02.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:02.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:02.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:02.008 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:02.008 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:02.008 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:02.009 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:02.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:02.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:02.009 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:02.009 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:02.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:02.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:02.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:02.010 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:02.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:02.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:02.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:02.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:02.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:02.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:02.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:02.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:02.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:02.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.013 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:02.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:02.013 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:02.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:02.014 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:02.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:02.014 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:02.015 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:07.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:07.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:07.018 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:07.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:07.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:07.434 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:07.435 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:07.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.437 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:07.438 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:07.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:07.440 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:07.441 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:07.443 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:07.444 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:12.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:12.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:12.448 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:12.488 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:12.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:12.864 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:12.864 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:12.864 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:12.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:12.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:12.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:12.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:12.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:12.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:12.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:12.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:12.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:12.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:12.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:12.871 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:12.872 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:12.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:12.874 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:12.875 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:17.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:17.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:17.878 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:17.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:17.917 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:18.284 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:18.284 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:18.285 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:18.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:18.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:18.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:18.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:18.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:18.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:18.293 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:18.294 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:23.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:23.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:23.297 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:23.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:23.337 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:23.709 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:23.710 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:23.711 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:23.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:23.713 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:23.714 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:23.715 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:23.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:23.717 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:23.718 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:23.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:23.720 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:23.721 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:28.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:28.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:28.723 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:28.768 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:28.768 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:29.135 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:29.135 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:29.135 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:29.135 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:29.136 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:29.137 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:29.138 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:29.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:29.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:29.141 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:29.142 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:29.143 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:29.144 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:29.144 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:29.145 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:34.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:34.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:34.148 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:34.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:34.188 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:34.552 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:34.552 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:34.552 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:34.553 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:34.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:34.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:34.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:34.557 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:34.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:34.557 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:34.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:34.558 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:34.559 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:34.559 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:34.559 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:34.559 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:34.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.560 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:34.560 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:34.560 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:34.560 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:34.560 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:34.561 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:34.561 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:34.561 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:34.561 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:34.562 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:34.562 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:34.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.562 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.563 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:34.563 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:34.563 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:34.563 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:34.563 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:34.564 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:34.564 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:34.565 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:39.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:39.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:39.568 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:39.608 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:39.608 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:39.869 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:39.869 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:39.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:39.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:39.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:39.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:39.874 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:39.875 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:39.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:39.876 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:39.876 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:39.877 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:39.877 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:39.877 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:39.879 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:39.880 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:44.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:44.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:44.883 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:44.924 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:44.924 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:45.285 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:45.285 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:45.285 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:45.286 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:45.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:45.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:45.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:45.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:45.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:45.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:45.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:45.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:45.292 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:45.293 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.294 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:45.294 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:45.294 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:45.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:45.295 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:45.295 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:45.295 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:45.297 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:50.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:50.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:50.300 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:50.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:50.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:50.702 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:50.703 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:50.703 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:50.703 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:50.703 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:50.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:50.705 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:50.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:50.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:50.706 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:50.706 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:50.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:50.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:50.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:50.707 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:50.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:50.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:50.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:50.708 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:50.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:50.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:50.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:50.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:50.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:50.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:50.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:50.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:50.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:50.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:50.712 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:50.712 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:50.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:50.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:50.713 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:50.713 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:50.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:50.713 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:50.714 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:50.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:50.715 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:50.716 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:52:55.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:55.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:52:55.720 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:52:55.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:52:55.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:52:56.125 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:52:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:52:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:52:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:52:56.125 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:56.126 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:56.127 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:56.128 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:56.130 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:52:56.131 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:52:56.132 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:52:56.133 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:52:56.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:52:56.134 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:52:56.135 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:01.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:01.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:01.137 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:01.179 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:01.180 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:01.537 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:01.538 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:01.539 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.540 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:01.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:01.542 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:01.543 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:01.544 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:01.546 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:01.547 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:06.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:06.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:06.549 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:06.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:06.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:06.966 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:06.966 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:06.967 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:06.968 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:06.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:06.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:06.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:06.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:06.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:06.974 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:06.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:06.975 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:06.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:06.975 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:06.976 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:11.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:11.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:11.979 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:12.020 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:12.020 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:12.389 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:12.389 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:12.390 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:12.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:12.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:12.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:12.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:12.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:12.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.397 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:12.398 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:12.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:17.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:17.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:17.402 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:17.444 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:17.444 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:17.812 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:17.813 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:17.813 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:17.813 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:17.813 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:17.813 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:17.814 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:17.815 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:17.816 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:17.817 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:17.818 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:17.819 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:17.820 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:17.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:17.822 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:17.823 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:22.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:22.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:22.825 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:22.866 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:22.866 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:23.233 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:23.233 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:23.234 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:23.234 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:23.234 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:23.235 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:23.235 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:23.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:23.235 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:23.236 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:23.236 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:23.236 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:23.237 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:23.237 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:23.237 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:23.237 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:23.238 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:23.238 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:23.238 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:23.238 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:23.239 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:23.239 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:23.239 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:23.240 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:23.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:23.240 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:23.241 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:23.241 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:23.241 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:23.241 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:23.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:23.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:23.242 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:23.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:23.243 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:23.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:23.244 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:23.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:23.244 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:23.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:23.245 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:23.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.245 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:23.245 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:23.246 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:23.246 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:23.246 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:23.247 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:23.247 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:23.247 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:23.247 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:23.248 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:23.248 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:23.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.249 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:23.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:23.249 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:23.250 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:23.250 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:23.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:23.250 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:23.252 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:28.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:28.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:28.254 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:28.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:28.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:28.663 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:28.664 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:28.665 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:28.666 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:28.667 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:28.668 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:28.669 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.670 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:28.671 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:28.672 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:28.673 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:28.673 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:28.674 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:33.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:33.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:33.677 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:33.718 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:33.718 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:33.986 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:33.987 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:33.987 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:33.987 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:33.988 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:33.988 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:33.988 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:33.989 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:33.989 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:33.989 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:33.989 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:33.990 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:33.991 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:33.992 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:33.993 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:33.994 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:33.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:33.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:33.997 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:33.998 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:33.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:34.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:34.001 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:34.001 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:34.002 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:34.002 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:34.002 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:34.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:34.003 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:34.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:34.003 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:34.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:34.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:34.004 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:34.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:34.005 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:34.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:34.005 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:34.006 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:34.006 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:34.007 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:39.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:39.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:53:39.011 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:53:39.053 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:39.053 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:39.420 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:53:39.421 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:53:39.421 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:53:39.421 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:53:39.421 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:53:39.421 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:53:39.422 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:53:39.423 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:39.424 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:39.425 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:39.426 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:53:39.427 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:53:39.428 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:53:39.429 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:53:39.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:53:39.430 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:53:39.431 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602265698.570765_374.1 target=1_1602265031.065489_324.1 2020-10-09T17:53:44.432 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... FAIL 2020-10-09T17:53:44.433 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-65 2020-10-09T17:53:44.482 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=wypfiu-66 2020-10-09T17:53:44.561 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=wypfiu-67 2020-10-09T17:53:46.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:53:46.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:53:46.388 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:53:46.438 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:53:46.438 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:53:46.688 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:53:46.683+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:53:46.689 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:53:46.689 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:53:51.692 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:53:51.687+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:53:51.693 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:53:51.693 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:53:56.696 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:53:56.691+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:53:56.697 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:53:56.697 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:01.701 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:01.695+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:54:01.702 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:54:01.702 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:06.704 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:06.699+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:54:06.705 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:54:06.705 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:11.708 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:11.703+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:54:11.709 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:54:11.709 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:16.712 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:16.707+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:54:16.713 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:54:16.713 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:21.716 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:21.711+0000 7f1908ac90c0 1 waiting for incremental sync to catch up: 2020-10-09T17:54:21.716 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000003.63.5, , , , , ] 2020-10-09T17:54:21.717 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:26.720 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:26.718+0000 7f1908ac90c0 1 bucket sync caught up with source: 2020-10-09T17:54:26.720 INFO:teuthology.orchestra.run.smithi184.stderr: local status: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:26.721 INFO:teuthology.orchestra.run.smithi184.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:26.721 INFO:teuthology.orchestra.run.smithi184.stderr:2020-10-09T17:54:26.718+0000 7f1908ac90c0 0 bucket checkpoint complete 2020-10-09T17:54:26.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:54:26.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:54:26.752 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket wypfiu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2020-10-09T17:54:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:26.805 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:27.052 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:54:27.047+0000 7fc3731af0c0 1 bucket sync caught up with source: 2020-10-09T17:54:27.053 INFO:teuthology.orchestra.run.smithi060.stderr: local status: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:27.053 INFO:teuthology.orchestra.run.smithi060.stderr: remote markers: [, , , , , 00000000011.71.5, , , , , ] 2020-10-09T17:54:27.053 INFO:teuthology.orchestra.run.smithi060.stderr:2020-10-09T17:54:27.047+0000 7fc3731af0c0 0 bucket checkpoint complete 2020-10-09T17:54:27.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:54:27.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'wypfiu-65', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2020-10-09T17:54:27.082 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket wypfiu-65 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2020-10-09T17:54:27.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:27.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:27.364 INFO:teuthology.orchestra.run.smithi184.stderr:could not get bucket info for bucket=wypfiu-65 2020-10-09T17:54:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-10-09T17:54:27.501 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... FAIL 2020-10-09T17:54:27.501 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-68 2020-10-09T17:54:27.626 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:54:27.623+0000 7f0cb6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:54:27.627 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:54:27.622+0000 7fd91b7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:54:27.627 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:54:27.623+0000 7f0c9dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:54:27.628 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:54:27.626+0000 7fd91aff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:54:27.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:27.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:27.635 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:27.671 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:27.672 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:27.866 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:54:27.894 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:54:27.894 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:27.894 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:54:27.894 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:27.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:27.896 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:54:27.896 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:54:27.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:27.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:27.897 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266067.540286_391.1", 2020-10-09T17:54:27.897 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:54:27.540286Z" 2020-10-09T17:54:27.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:27.897 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:54:27.899 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:54:27.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:27.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:27.901 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:27.944 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:27.945 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:28.189 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:28.190 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.191 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.193 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.194 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:28.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.196 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602266067.540286_391.1", 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:54:27.540286Z", 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.197 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.198 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:28.199 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:54:28.200 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:54:28.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:28.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:28.201 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:28.244 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:28.245 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:28.508 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:54:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:54:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:54:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:54:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:54:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266067.540286_391.1", 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:54:27.540286Z", 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:54:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:54:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:54:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:54:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:54:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:54:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:54:28.519 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:54:28.520 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:54:28.521 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:54:28.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:28.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:28.523 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:28.564 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:28.564 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:28.931 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:28.932 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:28.932 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:28.932 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:28.933 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:28.933 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:28.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:28.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.934 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:28.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:28.935 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.935 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.935 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:28.935 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.936 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:28.937 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.937 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.937 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.938 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.938 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:28.938 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.939 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.939 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:28.940 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.940 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.940 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.940 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:28.941 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.941 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.941 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.942 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.942 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:28.942 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.943 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.943 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:28.943 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.943 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.944 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.944 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:28.944 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:28.945 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.945 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:28.945 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:28.946 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:28.947 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:28.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:28.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.948 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.949 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:28.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:28.949 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:28.949 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:28.950 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:28.950 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:28.952 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:54:33.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:33.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:33.956 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:33.995 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:33.996 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:34.361 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:34.361 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:34.362 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:34.363 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:34.364 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:34.365 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:34.366 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:34.367 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:34.368 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:34.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:34.370 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:34.370 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:34.371 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:54:39.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:39.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:39.378 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:39.430 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:39.431 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:39.792 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:39.792 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:39.792 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:39.792 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:39.792 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:39.793 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:39.794 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:39.795 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:39.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:39.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:39.798 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:39.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:39.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:39.800 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:39.800 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:39.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:39.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:39.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:39.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:39.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.802 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:39.803 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:39.805 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:54:44.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:44.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:44.808 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:44.851 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:44.851 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:45.217 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:45.217 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:45.218 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:45.218 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:45.218 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:45.218 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:45.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:45.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:45.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:45.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:45.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:45.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:45.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.226 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:45.227 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:45.228 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:54:50.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:50.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:50.231 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:50.272 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:50.273 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:50.641 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:50.642 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:50.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:50.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:50.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:50.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:50.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:50.649 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:50.650 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:50.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:50.652 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:50.653 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:54:55.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:55.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:54:55.655 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:54:55.697 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:54:55.698 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:54:56.067 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:54:56.067 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:56.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:54:56.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:56.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:56.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:54:56.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:54:56.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:54:56.074 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.075 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:54:56.076 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:54:56.077 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:01.078 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:01.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:01.080 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:01.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:01.122 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:01.390 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:01.390 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:01.390 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:01.390 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:01.391 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:01.392 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:01.393 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:01.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:01.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:01.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:01.394 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:01.394 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:01.395 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:01.396 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:01.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:01.399 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:01.400 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:06.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:06.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:06.404 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:06.447 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:06.447 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:06.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:06.799 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:06.799 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:06.800 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:06.801 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:06.802 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:06.803 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:06.804 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:06.805 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:06.806 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:06.807 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:06.808 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:06.808 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:06.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:06.808 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:06.809 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:11.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:11.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:11.812 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:11.852 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:11.852 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:12.225 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:12.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:12.227 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:12.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:12.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:12.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:12.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:12.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:12.234 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:12.235 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:12.236 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:17.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:17.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:17.238 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:17.278 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:17.279 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:17.640 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:17.640 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:17.640 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:17.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:17.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:17.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:17.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:17.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:17.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:17.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.648 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:17.649 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:17.650 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:22.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:22.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:22.653 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:22.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:22.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:23.062 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:23.062 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:23.063 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:23.063 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:23.063 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:23.063 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:23.064 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:23.064 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:23.064 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:23.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:23.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:23.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:23.065 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:23.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:23.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:23.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:23.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:23.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:23.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:23.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:23.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:23.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:23.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:23.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:23.069 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:23.069 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:23.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:23.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:23.070 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:23.070 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:23.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:23.071 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:23.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:23.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:23.072 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:23.072 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:23.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:23.075 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:23.076 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:23.077 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:23.078 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:23.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:23.080 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:23.081 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:28.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:28.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:28.084 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:28.124 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:28.125 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:28.482 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:28.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:28.484 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:28.485 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:28.486 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:28.487 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:28.488 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:28.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.490 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:28.491 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:28.492 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:33.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:33.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:33.495 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:33.535 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:33.536 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:33.907 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:33.908 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:33.908 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:33.908 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:33.908 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:33.908 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:33.909 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:33.910 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:33.910 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:33.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:33.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:33.910 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:33.911 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:33.912 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:33.913 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:33.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:33.913 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:33.913 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:33.914 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:33.915 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:33.916 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:33.917 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:33.918 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:33.918 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:33.918 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:33.918 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:33.919 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:38.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:38.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:38.922 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:38.962 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:38.963 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:39.322 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:39.322 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:39.323 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:39.324 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:39.325 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:39.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:39.327 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:39.328 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:39.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:39.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:39.331 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:39.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:39.332 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:39.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:44.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:44.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:44.336 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:44.377 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:44.378 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:44.741 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:44.741 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:44.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:44.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:44.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:44.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:44.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:44.747 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:44.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:44.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:44.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:44.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:44.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:44.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:44.750 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:44.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:44.751 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:44.752 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:49.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:49.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:49.754 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:49.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:49.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:50.158 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:50.158 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:50.159 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:50.159 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:50.159 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:50.160 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:50.160 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:50.160 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:50.161 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:50.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:50.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:50.162 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:50.162 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:50.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:50.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:50.163 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:50.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:50.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:50.164 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:50.165 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.165 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:50.165 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:50.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:50.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:50.166 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:50.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:50.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:50.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:50.168 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:50.168 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:50.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:50.169 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.169 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:50.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:50.170 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:50.170 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:50.170 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:50.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:50.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:50.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:50.172 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:50.172 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:50.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:50.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.173 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:50.174 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:50.174 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:50.174 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:50.175 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:50.175 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:50.175 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:50.176 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:50.176 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:50.176 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:50.177 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:50.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.177 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.178 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:50.178 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:50.178 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:50.179 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:50.179 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:50.179 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:50.180 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:50.182 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:55:55.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:55.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:55:55.188 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:55:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:55:55.229 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:55:55.596 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:55:55.596 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:55:55.596 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:55:55.597 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:55:55.598 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:55.599 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:55.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:55:55.601 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:55.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:55:55.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:55:55.604 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:55:55.605 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:55:55.605 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:55:55.605 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:55:55.606 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:00.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:00.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:00.608 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:00.648 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:00.649 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:01.021 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:01.021 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:01.021 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:01.022 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:01.022 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:01.022 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:01.022 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:01.023 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:01.023 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:01.023 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:01.023 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:01.024 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:01.024 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:01.024 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:01.024 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:01.024 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:01.025 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:01.025 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:01.025 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:01.025 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.025 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:01.026 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:01.027 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:01.028 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:01.029 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:01.030 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:01.031 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:01.032 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:01.032 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:01.032 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:01.033 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:06.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:06.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:06.036 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:06.077 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:06.078 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:06.349 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:06.349 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:06.349 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:06.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:06.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:06.352 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:06.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:06.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:06.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:06.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:06.357 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:06.358 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:06.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:06.358 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:06.359 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:11.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:11.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:11.361 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:11.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:11.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:11.768 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:11.768 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:11.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:11.770 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:11.771 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:11.772 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:11.773 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:11.774 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:11.775 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:11.776 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:11.777 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:11.778 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:11.779 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:16.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:16.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:16.781 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:16.822 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:16.823 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:17.187 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:17.187 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:17.188 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:17.189 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:17.190 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:17.191 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:17.192 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:17.193 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:17.194 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.195 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:17.196 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:17.197 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:17.198 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:17.199 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:17.199 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:17.200 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:22.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:22.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:22.204 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:22.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:22.243 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:22.607 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:22.607 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:22.607 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:22.607 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:22.608 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:22.608 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:22.608 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:22.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:22.609 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:22.609 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:22.609 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:22.609 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:22.609 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:22.610 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:22.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:22.610 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:22.610 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:22.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:22.612 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:22.613 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:22.614 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:22.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:22.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:22.615 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.615 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:22.615 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:22.616 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:22.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:22.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:22.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:22.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.618 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:22.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:22.618 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:22.618 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:22.619 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:22.619 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:22.619 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:22.620 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:27.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:27.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:27.623 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:27.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:27.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:28.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:28.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:28.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:28.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:28.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:28.035 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:28.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:28.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:28.039 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:28.040 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.041 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:28.042 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:28.043 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:28.044 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.045 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:28.046 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:28.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:28.047 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:28.048 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:33.049 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:33.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:33.051 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:33.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:33.094 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:33.457 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:33.457 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:33.457 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:33.458 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:33.459 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:33.460 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:33.460 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:33.460 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:33.460 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:33.461 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:33.461 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:33.461 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:33.461 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:33.462 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:33.462 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.462 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:33.462 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:33.462 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:33.463 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:33.464 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:33.464 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:33.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:33.465 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:33.465 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:33.465 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:33.465 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:33.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:33.466 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:33.467 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.468 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:33.469 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:33.471 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:38.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:38.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:38.474 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:38.515 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:38.515 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:38.879 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:38.880 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:38.880 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:38.880 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:38.880 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:38.880 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:38.881 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:38.882 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:38.883 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:38.884 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:38.885 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:38.885 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:38.885 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:38.886 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:38.886 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.886 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:38.886 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:38.887 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:38.887 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:38.887 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:38.888 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:38.888 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:38.888 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:38.889 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:38.889 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:38.889 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:38.889 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.890 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:38.890 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:38.890 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:38.891 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:38.891 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:38.892 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:38.892 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:38.892 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:38.893 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:38.894 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:38.895 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:38.895 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:38.895 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:38.896 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:38.897 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:43.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:43.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:43.901 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:43.940 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:43.941 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:44.301 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:44.301 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:44.301 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:44.302 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:44.302 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:44.302 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:44.303 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:44.303 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:44.303 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:44.303 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:44.304 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:44.304 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:44.304 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:44.305 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:44.305 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:44.305 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:44.306 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:44.306 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:44.306 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:44.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:44.307 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:44.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:44.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:44.309 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:44.309 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:44.311 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:44.312 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:44.313 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:44.314 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:44.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:44.315 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:44.316 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:49.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:49.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:49.319 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:49.359 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:49.360 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:49.628 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:49.628 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:49.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:49.630 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:49.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:49.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:49.633 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:49.634 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:49.635 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:49.636 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:49.637 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:49.638 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:49.638 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:49.638 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:49.639 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:56:54.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:54.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:56:54.643 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:56:54.718 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:56:54.718 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:56:55.083 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:56:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:56:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:56:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:56:55.083 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:56:55.084 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:56:55.085 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:55.086 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.087 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:55.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:56:55.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:56:55.090 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:56:55.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:56:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:56:55.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:56:55.092 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:56:55.093 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:00.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:00.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:00.096 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:00.140 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:00.140 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:00.493 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:00.494 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:00.494 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:00.495 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:00.495 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:00.495 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:00.495 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:00.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:00.496 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:00.496 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:00.496 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:00.497 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:00.497 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:00.498 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:00.498 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:00.498 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:00.499 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:00.499 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:00.499 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:00.499 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.500 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:00.500 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:00.500 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:00.500 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:00.501 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:00.501 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:00.501 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:00.502 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:00.502 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:00.502 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:00.502 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:00.503 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.503 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:00.503 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:00.504 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:00.504 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:00.504 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:00.505 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:00.505 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:00.505 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:00.506 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:00.506 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:00.506 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:00.507 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:00.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:00.509 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.510 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:00.511 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:00.513 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:05.514 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:05.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:05.515 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:05.558 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:05.558 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:05.927 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:05.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:05.927 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:05.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:05.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:05.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:05.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:05.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:05.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:05.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:05.935 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:05.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:05.936 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:05.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:05.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:05.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:10.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:10.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:10.940 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:10.981 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:10.981 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:11.250 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:11.250 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:11.251 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:11.252 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:11.253 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:11.254 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:11.255 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:11.255 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:11.255 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:11.256 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:11.256 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.256 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:11.256 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:11.257 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.258 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:11.260 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:11.261 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:11.262 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:11.263 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:11.263 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:11.264 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:16.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:16.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:16.267 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:16.310 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:16.311 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:16.672 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:16.672 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:16.673 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:16.674 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:16.675 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:16.676 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:16.677 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:16.679 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:16.680 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:16.681 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:16.681 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:16.681 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:16.681 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:16.681 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:16.682 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:21.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:21.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:21.684 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:21.726 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:21.726 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:22.093 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:22.093 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:22.094 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:22.095 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:22.096 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:22.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:22.098 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:22.099 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:22.100 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:22.101 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:22.101 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.102 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:22.103 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:22.104 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:27.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:27.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:27.107 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:27.151 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:27.152 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:27.411 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:27.412 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:27.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:27.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:27.416 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:27.416 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:27.416 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:27.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.417 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:27.417 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:27.417 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:27.417 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:27.417 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:27.418 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:27.418 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:27.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:27.418 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:27.419 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:27.419 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:27.419 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.419 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:27.420 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:27.420 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:27.420 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:27.420 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:27.421 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:27.422 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:27.423 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:27.423 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:27.424 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:32.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:32.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:32.426 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:32.466 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:32.466 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:32.839 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:32.840 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:32.841 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.842 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:32.843 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:32.844 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:32.845 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:32.846 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:32.847 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:32.848 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:32.849 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:32.850 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:37.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:37.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:37.853 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:37.906 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:37.907 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:38.274 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:38.274 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:38.274 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:38.275 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:38.276 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:38.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:38.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:38.276 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:38.276 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:38.277 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:38.278 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:38.279 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:38.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:38.281 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:38.282 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:38.283 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:38.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:38.284 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:38.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:38.284 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:38.285 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:43.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:43.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:43.287 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:43.328 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:43.329 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:43.693 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:43.693 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:43.693 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:43.693 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:43.694 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:43.695 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:43.696 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:43.697 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:43.698 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:43.698 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.698 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:43.698 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:43.699 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:43.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:43.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:43.702 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:43.703 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:43.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:43.703 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:43.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:48.705 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:48.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:48.706 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:48.747 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:48.747 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:49.010 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:49.010 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:49.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:49.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:49.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:49.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:49.015 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:49.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:49.018 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:49.019 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:49.020 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:49.021 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:49.021 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:49.022 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:49.023 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:54.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:54.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:54.025 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:54.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:54.065 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:54.437 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:54.438 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:54.438 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:54.438 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:54.439 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:54.439 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:54.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:54.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:54.440 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:54.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:54.440 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:54.441 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:54.441 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:54.441 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:54.441 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:54.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:54.442 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:54.442 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:54.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:54.443 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.443 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:54.443 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:54.444 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:54.444 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:54.444 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:54.444 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:54.445 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:54.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:54.445 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:54.445 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:54.446 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:54.447 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:54.448 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.449 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:54.450 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:54.451 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:54.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:57:59.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:59.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:57:59.453 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:57:59.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:57:59.494 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:57:59.858 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:57:59.858 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:57:59.858 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:57:59.858 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:59.859 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:59.860 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:59.861 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:59.862 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:57:59.863 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:57:59.864 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:57:59.865 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:57:59.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:57:59.867 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:57:59.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:57:59.867 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:57:59.868 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:04.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:04.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:04.871 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:04.913 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:04.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:05.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:05.284 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:05.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:05.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:05.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:05.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:05.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.291 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:05.292 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:05.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:10.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:10.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:10.296 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:10.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:10.338 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:10.805 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:10.806 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:10.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:10.808 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:10.809 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:10.810 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.811 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:10.812 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:10.813 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:10.814 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:10.815 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:10.816 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:10.817 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.818 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:10.819 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:10.820 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:10.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:10.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:10.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:10.824 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:10.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:15.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:15.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:15.828 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:15.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:15.865 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:16.226 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:16.233 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:16.234 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:16.235 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:16.236 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:16.236 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:16.237 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:21.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:21.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:21.239 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:21.279 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:21.280 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:21.638 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:21.641 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:21.641 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:21.641 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:21.641 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:21.641 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:21.642 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:21.643 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:21.644 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:21.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:21.650 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:21.651 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:26.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:26.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:26.654 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:26.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:26.693 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:27.061 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:27.062 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:27.062 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:27.062 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:27.062 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:27.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:27.074 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:27.075 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:32.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:32.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:32.077 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:32.110 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:32.111 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:32.472 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:32.473 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:32.474 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:32.475 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:32.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:32.478 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:32.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:32.480 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:32.481 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:32.482 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:32.482 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:32.482 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:32.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:32.482 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:32.483 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:37.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:37.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:37.485 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:37.525 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:37.526 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:37.893 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:37.894 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:37.894 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:37.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:37.894 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:37.894 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:37.895 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:37.896 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:37.897 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:37.898 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:37.899 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:37.900 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:37.901 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:37.902 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:37.903 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:37.904 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:37.904 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:37.905 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:42.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:42.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:42.908 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:42.951 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:42.951 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:43.317 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:43.318 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:43.319 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:43.320 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:43.321 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:43.322 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.323 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:43.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:43.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:43.326 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:43.327 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:43.328 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:43.329 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:43.330 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:43.331 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:43.332 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.333 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:43.334 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:43.335 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:43.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:48.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:48.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:48.339 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:48.380 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:48.380 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:48.747 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:48.747 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:48.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:48.749 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:48.750 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:48.751 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:48.751 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:48.751 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:48.751 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:48.752 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:48.753 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:48.754 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:48.755 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.756 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.756 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:48.756 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:48.756 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:48.756 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:48.757 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:48.757 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:48.757 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:48.757 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:53.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:53.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:53.760 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:53.804 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:53.830 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:54.167 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:54.205 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:54.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:54.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:54.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:54.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:54.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:54.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:54.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:54.214 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:54.214 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:54.215 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:58:59.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:59.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:58:59.218 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:58:59.262 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:58:59.262 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:58:59.632 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:58:59.632 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:58:59.632 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:58:59.632 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:58:59.633 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:58:59.633 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:58:59.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:59.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:59.633 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:58:59.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:59.634 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:58:59.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:59.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:59.635 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:58:59.635 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:59.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:58:59.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:59.636 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:58:59.636 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:59.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.638 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:59.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:59.638 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:58:59.638 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:59.639 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:59.639 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:58:59.639 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.640 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:58:59.641 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:59.642 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:58:59.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:58:59.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:58:59.645 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:58:59.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:58:59.645 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:58:59.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:58:59.645 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:58:59.646 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:04.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:04.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:04.649 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:04.691 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:04.691 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:05.056 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:05.057 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:05.057 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:05.057 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:05.057 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:05.058 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:05.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:05.058 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:05.058 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:05.058 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:05.059 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:05.060 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:05.061 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:05.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:05.063 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:05.064 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:05.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:05.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:05.067 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:05.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:05.068 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:05.068 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:05.068 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:05.070 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:10.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:10.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:10.075 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:10.116 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:10.116 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:10.471 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:10.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:10.473 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:10.474 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:10.475 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.476 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:10.477 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:10.478 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:10.478 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:10.478 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:10.479 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:10.479 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:10.479 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.479 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:10.479 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:10.480 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:10.480 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:10.480 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:10.480 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:10.481 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:10.481 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:10.481 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:10.481 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:10.482 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:10.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.482 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.483 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:10.483 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:10.483 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:10.484 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:10.484 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:10.484 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:10.484 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:10.487 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:15.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:15.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:15.490 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:15.534 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:15.534 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:15.904 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:15.905 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:15.906 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:15.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:15.908 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.909 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:15.910 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:15.911 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:15.912 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:15.913 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:15.914 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:15.914 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:15.915 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:20.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:20.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:20.918 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:20.957 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:20.957 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:21.321 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:21.321 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:21.321 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:21.321 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:21.322 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:21.323 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:21.324 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:21.325 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:21.326 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:21.327 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.328 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:21.329 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:21.329 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:21.329 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:21.329 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:21.329 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:21.330 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:21.331 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:21.332 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:21.332 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:21.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:26.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:26.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:26.336 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:26.375 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:26.376 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:26.745 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:26.745 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:26.746 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:26.747 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:26.748 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:26.749 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:26.750 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:26.751 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:26.752 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:26.753 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:26.754 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:26.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:31.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:31.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:31.757 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:31.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:31.798 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:32.165 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:32.165 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:32.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:32.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:32.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:32.170 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:32.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:32.172 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:32.173 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:32.174 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:32.175 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:32.176 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:37.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:37.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:37.178 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:37.216 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:37.216 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:37.581 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:37.581 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:37.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:37.583 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:37.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:37.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:37.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:37.588 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.589 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:37.590 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:37.591 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:42.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:42.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:42.594 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:42.638 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:42.638 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:43.009 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:43.009 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:43.010 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:43.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:43.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:43.013 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:43.014 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.015 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:43.016 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:43.017 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:43.018 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:43.018 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:43.018 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:43.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:43.018 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:43.019 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:48.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:48.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:48.022 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:48.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:48.061 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:48.433 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:48.433 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:48.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:48.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:48.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:48.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:48.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:48.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:48.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:48.440 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:48.440 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:48.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:48.441 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:48.441 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:48.441 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:48.442 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:48.442 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:48.442 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:48.443 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:48.443 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:48.443 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:48.443 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:48.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.444 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.444 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:48.445 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:48.445 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:48.445 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:48.445 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:48.446 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:48.446 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:48.447 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266067.540286_391.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:53.449 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... FAIL 2020-10-09T17:59:53.450 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=wypfiu-69 2020-10-09T17:59:53.602 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:59:53.600+0000 7f0c9e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:59:53.603 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-10-09T17:59:53.600+0000 7f0c967fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:59:53.605 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:59:53.600+0000 7fd9277fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:59:53.605 INFO:tasks.rgw.c2.client.0.smithi184.stdout:2020-10-09T17:59:53.600+0000 7fd91dffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-10-09T17:59:53.706 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:53.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:53.708 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:53.746 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:53.746 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:53.927 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T17:59:53.961 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T17:59:53.961 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:59:53.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T17:59:53.963 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:59:53.490902Z" 2020-10-09T17:59:53.964 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:53.964 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T17:59:53.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T17:59:53.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:53.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:53.966 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:54.008 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:54.008 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:54.239 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:54.240 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:54.240 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:54.240 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:54.240 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:54.240 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.241 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:54.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.244 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.245 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.246 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:59:53.490902Z", 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:54.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.248 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.248 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:54.248 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.248 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:54.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:54.249 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:54.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.249 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:54.250 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T17:59:54.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T17:59:54.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:54.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:54.251 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:54.289 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:54.290 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:54.536 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T17:59:54.536 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T17:59:54.536 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T17:59:54.536 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:54.537 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:59:54.538 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:59:54.539 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:59:54.540 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:54.541 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:54.542 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T17:59:54.543 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:59:53.490902Z", 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T17:59:54.544 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T17:59:54.545 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T17:59:54.545 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T17:59:54.545 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T17:59:54.546 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T17:59:54.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T17:59:54.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:54.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:54.547 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:54.588 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:54.589 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T17:59:54.864 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T17:59:54.864 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T17:59:54.864 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T17:59:54.864 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T17:59:54.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T17:59:54.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T17:59:54.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.869 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.870 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T17:59:54.871 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T17:59:54.872 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T17:59:54.873 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T17:59:54.873 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T17:59:54.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T17:59:54.873 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T17:59:54.874 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T17:59:59.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:59.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T17:59:59.878 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T17:59:59.926 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T17:59:59.927 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:00.293 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:00.293 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:00.294 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:00.294 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:00.294 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:00.294 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:00.295 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:00.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:00.295 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:00.295 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:00.296 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:00.296 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:00.296 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:00.296 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:00.297 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:00.298 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:00.298 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:00.298 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:00.299 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:00.300 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:00.300 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:00.300 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:00.300 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:00.300 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:00.301 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:00.301 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:00.301 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.301 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:00.302 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:00.303 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:00.303 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:00.303 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:00.303 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:00.304 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:00.304 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:00.304 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:00.304 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:00.305 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:00.305 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:00.305 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:00.306 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:00.306 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:00.307 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:00.307 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:00.307 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:00.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:00.307 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:00.308 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:00.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:00.309 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:00.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:00.309 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:00.310 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:05.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:05.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:05.313 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:05.352 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:05.353 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:05.715 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:05.715 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:05.716 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:05.717 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:05.718 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:05.719 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:05.720 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:05.721 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:05.722 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:05.723 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:05.724 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:05.724 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.725 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:05.726 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:05.727 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:10.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:10.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:10.730 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:10.771 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:10.771 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:11.130 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:11.131 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:11.131 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:11.132 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:11.132 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:11.132 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:11.133 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:11.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:11.133 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:11.134 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:11.134 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:11.134 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:11.135 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:11.136 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:11.137 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:11.138 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:11.139 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:11.140 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:11.141 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:11.142 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:11.142 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:11.142 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:11.142 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:11.142 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:11.143 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:16.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:16.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:16.145 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:16.186 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:16.187 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:16.552 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:16.552 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:16.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:16.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:16.555 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:16.556 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:16.557 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:16.558 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:16.559 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:16.560 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:16.561 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:16.562 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:16.563 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:21.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:21.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:21.565 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:21.606 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:21.606 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:21.870 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:21.870 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:21.870 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:21.871 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:21.872 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:21.873 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:21.874 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:21.875 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.876 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:21.877 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:21.878 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:21.878 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:21.878 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:21.878 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:21.879 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:21.880 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:21.880 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:21.881 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:26.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:26.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:26.883 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:26.920 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:26.921 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:27.290 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:27.308 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:27.309 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:27.309 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:27.309 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:27.309 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:27.310 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:27.310 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:27.310 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:27.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:27.310 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:27.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:27.312 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:27.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:27.314 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:27.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.316 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:27.316 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:27.316 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:27.316 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:27.316 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:27.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:27.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:27.319 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:27.320 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:32.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:32.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:32.323 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:32.364 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:32.364 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:32.736 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:32.737 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:32.738 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:32.739 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:32.740 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:32.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:32.742 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:32.743 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:32.744 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:32.745 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.746 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:32.747 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:32.748 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:37.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:37.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:37.751 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:37.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:37.793 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:38.161 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:38.161 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:38.162 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:38.163 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:38.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:38.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:38.167 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:38.168 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:38.168 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.168 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:38.168 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:38.168 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:38.169 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.170 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:38.171 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:38.172 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:43.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:43.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:43.175 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:43.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:43.217 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:43.587 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:43.734 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:43.735 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:43.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:43.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:43.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:43.739 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.740 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:43.741 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:43.742 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:43.743 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:43.743 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:43.743 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:43.744 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:43.744 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:43.745 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:48.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:48.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:48.747 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:48.788 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:48.788 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:49.159 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:49.160 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:49.160 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:49.160 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:49.160 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:49.160 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:49.161 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:49.162 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:49.163 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:49.164 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:49.165 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:49.166 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:49.167 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:49.167 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:49.167 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:49.167 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:49.167 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:49.171 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:49.171 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:49.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.171 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:49.172 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:49.173 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:49.174 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:54.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:54.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:54.177 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:54.213 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:54.213 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:54.579 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:54.579 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:54.579 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:54.579 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:54.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:54.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:54.582 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:54.583 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:54.584 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.585 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:54.586 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:54.587 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.588 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:54.589 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:54.590 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:54.591 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:54.591 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:54.593 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:00:59.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:59.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:00:59.595 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:00:59.637 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:00:59.637 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:00:59.897 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:00:59.898 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:00:59.898 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:00:59.898 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:00:59.898 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:00:59.899 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:00:59.900 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:59.901 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:00:59.902 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:59.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.903 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:59.904 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:59.904 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:00:59.904 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:59.904 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:00:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:59.905 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:00:59.906 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:00:59.907 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:00:59.908 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:00:59.909 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:04.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:04.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:04.912 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:04.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:04.950 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:05.308 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:05.308 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:05.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:05.310 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:05.312 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:05.313 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:05.314 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:05.315 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.315 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:05.315 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:05.315 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:05.315 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:05.316 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:05.316 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:05.316 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:05.316 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:05.316 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:05.317 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:05.318 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:05.318 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:05.318 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:05.318 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:05.319 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:10.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:10.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:10.322 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:10.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:10.365 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:10.729 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:10.730 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:10.731 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:10.732 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:10.733 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:10.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:10.735 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:10.736 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:10.737 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.738 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:10.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:10.740 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:10.741 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:15.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:15.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:15.743 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:15.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:15.786 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:16.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:16.218 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:16.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:16.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:16.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:16.222 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:16.222 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:16.222 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:16.222 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.223 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:16.224 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:16.224 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:16.224 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:16.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:16.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:16.225 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:16.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:16.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:16.226 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:16.226 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:16.226 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:16.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:16.228 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:16.229 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.230 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:16.231 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:16.232 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:16.232 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:16.233 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:21.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:21.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:21.236 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:21.276 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:21.276 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:21.645 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:21.646 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:21.647 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:21.648 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:21.649 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:21.650 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:21.651 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:21.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:21.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:21.654 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:21.655 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:21.655 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:21.655 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:21.655 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:21.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:26.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:26.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:26.658 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:26.698 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:26.698 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:27.064 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:27.064 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:27.065 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:27.066 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:27.067 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:27.068 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:27.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:27.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:27.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:27.072 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:27.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:27.074 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:27.075 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:27.076 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:32.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:32.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:32.078 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:32.120 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:32.120 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:32.573 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:32.573 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:32.573 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:32.574 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:32.575 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:32.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:32.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:32.578 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:32.579 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:32.580 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:32.581 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:32.582 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:32.583 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:32.583 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:32.583 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:32.584 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:37.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:37.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:37.586 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:37.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:37.629 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:37.998 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:37.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:38.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.001 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:38.002 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:38.003 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:38.004 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:38.005 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:38.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.007 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:38.008 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:38.010 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:43.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:43.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:43.012 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:43.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:43.054 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:43.424 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:43.424 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:43.424 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:43.425 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:43.425 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:43.425 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:43.425 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:43.425 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:43.426 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:43.426 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:43.426 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:43.426 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:43.426 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:43.427 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:43.427 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:43.427 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:43.427 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:43.427 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:43.428 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:43.428 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.428 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:43.428 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:43.428 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:43.429 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:43.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:43.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:43.432 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:43.433 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:43.434 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:43.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:43.435 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:43.436 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:48.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:48.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:48.439 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:48.479 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:48.480 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:48.850 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:48.850 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:48.850 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:48.850 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:48.851 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:48.852 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:48.853 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.854 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:48.855 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:48.856 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:48.857 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:48.858 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:48.859 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:48.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:48.859 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:48.860 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:53.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:53.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:53.863 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:53.904 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:53.905 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:54.265 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:54.266 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:54.266 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:54.266 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:54.267 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:54.268 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:54.268 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:54.268 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:54.269 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:54.269 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:54.269 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:54.270 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:54.270 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:54.271 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:54.271 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:54.271 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:54.271 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.272 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:54.272 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:54.272 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:54.272 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:54.273 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:54.273 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:54.273 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:54.273 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:54.274 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:54.274 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:54.274 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:54.275 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.275 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:54.275 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:54.276 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:54.276 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:54.276 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:54.277 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:54.277 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:54.277 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:54.277 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:54.278 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:54.278 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:54.278 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.278 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:54.279 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:54.279 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:54.279 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:54.279 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:54.280 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:54.280 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:54.280 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:54.280 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:54.281 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.282 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.282 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:54.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:54.282 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:54.283 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:54.283 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:54.283 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:54.283 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:54.284 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:01:59.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:59.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:01:59.287 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:01:59.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:01:59.330 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:01:59.698 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:01:59.699 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:59.700 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:59.701 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:59.702 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:59.703 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:59.704 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:01:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:59.705 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:01:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:01:59.706 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:01:59.707 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.708 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:01:59.709 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:01:59.710 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:04.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:04.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:04.714 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:04.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:04.753 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:05.115 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:05.115 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:05.115 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:05.116 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:05.117 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:05.117 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:05.117 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:05.117 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:05.118 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:05.118 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:05.118 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:05.119 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:05.119 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:05.119 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:05.119 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:05.120 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:05.121 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:05.122 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:05.123 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:05.124 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:05.125 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:05.125 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:05.125 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:05.125 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.126 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.126 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:05.126 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:05.126 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:05.127 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:05.127 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:05.127 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:05.128 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:05.129 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:10.129 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:10.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:10.131 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:10.174 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:10.175 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:10.546 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:10.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:10.548 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.549 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:10.550 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:10.551 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:10.552 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:10.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:10.555 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:10.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:15.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:15.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:15.558 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:15.601 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:15.601 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:15.967 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:15.968 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:15.969 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.970 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:15.971 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:15.972 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:15.973 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:15.974 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.975 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:15.976 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:15.977 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:20.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:20.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:20.980 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:21.020 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:21.021 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:21.388 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:21.388 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:21.388 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:21.389 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:21.390 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:21.391 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:21.392 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:21.393 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:21.394 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:21.395 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:21.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:21.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:21.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.397 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:21.398 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:21.399 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:26.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:26.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:26.403 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:26.446 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:26.446 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:26.819 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:26.819 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:26.820 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:26.821 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:26.822 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:26.823 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:26.824 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:26.825 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:26.826 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:26.827 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:26.828 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:26.829 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:26.829 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:26.829 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:26.830 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:31.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:31.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:31.834 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:31.871 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:31.872 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:32.338 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:32.338 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:32.338 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:32.338 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:32.339 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:32.340 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:32.341 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:32.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:32.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:32.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:32.346 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:32.347 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:32.347 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:32.347 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:32.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:32.347 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:32.348 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:37.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:37.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:37.351 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:37.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:37.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:37.759 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:37.759 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:37.760 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:37.761 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:37.762 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:37.763 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:37.764 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:37.765 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:37.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:37.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:37.769 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:37.769 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:37.769 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:37.769 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:37.769 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:37.770 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:42.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:42.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:42.773 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:42.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:42.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:43.182 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:43.183 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:43.183 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:43.183 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:43.183 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:43.184 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:43.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:43.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:43.184 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:43.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:43.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:43.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:43.187 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:43.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:43.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:43.190 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:43.191 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:43.192 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:43.193 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:43.194 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:48.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:48.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:48.197 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:48.236 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:48.236 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:48.597 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:48.597 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:48.598 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:48.599 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.600 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:48.601 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:48.602 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:48.603 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:48.604 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:48.605 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:48.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:48.607 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:48.607 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:48.608 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:53.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:53.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:53.611 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:53.657 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:53.657 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:53.926 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:53.926 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:53.926 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:53.926 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:53.927 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:53.928 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:53.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:53.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:53.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:53.932 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:53.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.934 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:53.935 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:53.936 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:02:58.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:58.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:02:58.939 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:02:58.978 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:02:58.979 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:02:59.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:02:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:02:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:02:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:02:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:02:59.341 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:59.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:59.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:02:59.344 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:59.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:59.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:02:59.347 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:02:59.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:02:59.349 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:02:59.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:02:59.350 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:02:59.351 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:04.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:04.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:04.354 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:04.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:04.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:04.761 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:04.761 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:04.761 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:04.761 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:04.761 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:04.762 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:04.763 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:04.764 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:04.764 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.764 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:04.764 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:04.764 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:04.765 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:04.766 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:04.767 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:04.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:04.768 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:04.768 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:04.769 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:04.770 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:04.771 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:04.771 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:04.771 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:04.772 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:09.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:09.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:09.775 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:09.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:09.816 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:10.188 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:10.216 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:10.217 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:10.218 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:10.219 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:10.220 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:10.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:10.222 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:10.223 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:10.224 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:10.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:10.225 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:10.228 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:15.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:15.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:15.231 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:15.274 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:15.274 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:15.643 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:15.644 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:15.644 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:15.644 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:15.644 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:15.644 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:15.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:15.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:15.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:15.646 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:15.646 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:15.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:15.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:15.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:15.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.647 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:15.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:15.649 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:15.649 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:15.649 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:15.649 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.649 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:15.650 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:15.651 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:15.651 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:15.651 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:15.651 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:15.652 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:15.653 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:15.654 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:15.655 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:15.656 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:20.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:20.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:20.658 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:20.699 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:20.699 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:21.066 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:21.066 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:21.067 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:21.068 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:21.069 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:21.070 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:21.071 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:21.072 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:21.073 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:21.074 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:21.075 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:21.076 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:21.077 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:21.077 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:21.077 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:21.077 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:21.078 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:21.078 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:21.078 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:21.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:21.080 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:21.080 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:21.080 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:21.081 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:21.081 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:21.081 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:21.082 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:21.082 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:21.083 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:21.084 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:21.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:21.084 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:21.085 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:26.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:26.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:26.089 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:26.130 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:26.130 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:26.486 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:26.487 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:26.488 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:26.489 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:26.490 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.491 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:26.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:26.493 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:26.493 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:26.493 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:26.493 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.493 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:26.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:26.494 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:26.494 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:26.494 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:26.494 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.495 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:26.496 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:26.497 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:31.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:31.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:31.500 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:31.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:31.540 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:32.005 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:32.006 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:32.006 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:32.006 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:32.006 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:32.006 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:32.007 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:32.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:32.009 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.010 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:32.011 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:32.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:32.013 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:32.014 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:32.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:32.015 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:32.016 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:37.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:37.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:37.019 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:37.060 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:37.060 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:37.427 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:37.428 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:37.428 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:37.428 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:37.428 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:37.429 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:37.430 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:37.431 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:37.431 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:37.431 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:37.431 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:37.432 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:37.432 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.432 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:37.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:37.433 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:37.433 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:37.433 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:37.434 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:37.434 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:37.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:37.435 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:37.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:37.435 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:37.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.436 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:37.436 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:37.436 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:37.436 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:37.437 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:37.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:37.440 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:37.441 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:42.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:42.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:42.444 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:42.484 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:42.484 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:42.856 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:42.857 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:42.857 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:42.857 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:42.857 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:42.857 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:42.858 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:42.859 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:42.860 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:42.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:42.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:42.863 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:42.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:42.865 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:42.866 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:42.866 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:47.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:47.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:47.869 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:47.908 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:47.909 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:48.281 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:48.282 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:48.283 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:48.284 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:48.285 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:48.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:48.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:48.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.289 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:48.290 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:48.291 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:53.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:53.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:53.293 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:53.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:53.335 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:53.704 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:53.705 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:53.705 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:53.706 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:53.706 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:53.706 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:53.706 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:53.706 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:53.707 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:53.707 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:53.707 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:53.707 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:53.707 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:53.708 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:53.708 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:53.708 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:53.708 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:53.708 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:53.709 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:53.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:53.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:53.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:53.710 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:53.710 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:53.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:53.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:53.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:53.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:53.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:53.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:53.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:53.715 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:53.716 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:53.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:53.716 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:53.717 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:03:58.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:58.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:03:58.720 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:03:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:03:58.762 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:03:59.126 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:03:59.126 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:03:59.126 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:03:59.127 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:59.128 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:59.129 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:59.130 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:03:59.131 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:03:59.132 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:03:59.133 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:03:59.134 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:03:59.134 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:03:59.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:03:59.134 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:03:59.134 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.135 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:03:59.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:03:59.135 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:03:59.135 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:03:59.136 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:03:59.136 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:03:59.136 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:03:59.138 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:04.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:04.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:04.140 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:04.182 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:04.183 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:04.551 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:04.551 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:04.551 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:04.552 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:04.553 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:04.554 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:04.555 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:04.556 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:04.557 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:04.558 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:04.559 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:04.560 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:04.560 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:04.560 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:04.560 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:04.561 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:09.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:09.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:09.563 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:09.605 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:09.606 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:09.973 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:09.973 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:09.973 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:09.973 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:09.973 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:09.974 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:09.975 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:09.976 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:09.977 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:09.978 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:09.979 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:09.980 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:09.981 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:09.982 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:09.982 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:14.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:14.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:14.985 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:15.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:15.025 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:15.395 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:15.396 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:15.397 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.398 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:15.399 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:15.407 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:15.407 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:15.407 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:15.408 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:15.409 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:15.410 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.411 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:15.412 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:15.414 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:20.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:20.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:20.418 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:20.459 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:20.460 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:20.828 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:20.828 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:20.828 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:20.828 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:20.829 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:20.830 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:20.831 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:20.832 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:20.833 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:20.834 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:20.835 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:20.836 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:20.837 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:20.837 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:20.838 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:25.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:25.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:25.841 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:25.882 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:25.882 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:26.241 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:26.241 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:26.242 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:26.243 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:26.244 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:26.245 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:26.246 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.247 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:26.248 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:26.249 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:26.250 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:26.250 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:26.251 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:31.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:31.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:31.254 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:31.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:31.293 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:31.660 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:31.660 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:31.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:31.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:31.664 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:31.665 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:31.666 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:31.667 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:31.668 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:31.669 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:31.670 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:31.670 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:31.671 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:36.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:36.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:36.676 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:36.717 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:36.718 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:37.086 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:37.087 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:37.087 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:37.087 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:37.088 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:37.088 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:37.088 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:37.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:37.089 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:37.090 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:37.091 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:37.092 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:37.093 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:37.094 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:37.095 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:37.096 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:37.097 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:37.098 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:37.099 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:37.099 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:37.099 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:37.100 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:37.101 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:37.101 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:37.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.101 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.102 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:37.102 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:37.102 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:37.102 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:37.103 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:37.103 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:37.103 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:37.105 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:42.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:42.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:42.107 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:42.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:42.149 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:42.509 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:42.510 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:42.511 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:42.512 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:42.513 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.514 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:42.515 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:42.516 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:42.517 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:42.518 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:42.519 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:42.519 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:42.519 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:42.519 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:42.520 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:47.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:47.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:47.523 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:47.563 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:47.564 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:47.927 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:47.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:47.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:47.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:47.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:47.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:47.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:47.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:47.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:47.936 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:47.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:47.936 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:47.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:47.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:47.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:52.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:52.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:52.940 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:52.980 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:52.980 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:53.347 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:53.347 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:53.347 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:53.348 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:53.349 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:53.350 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:53.351 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:53.352 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:53.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:53.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:53.355 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:53.356 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:53.356 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:53.356 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:53.357 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:04:58.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:58.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:04:58.359 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:04:58.401 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:04:58.402 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:04:58.774 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:04:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:04:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:04:58.775 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:04:58.776 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:58.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:04:58.777 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:58.777 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:58.777 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:04:58.778 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:58.778 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:04:58.778 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:58.778 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:04:58.779 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:58.779 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.779 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:58.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:58.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:04:58.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:58.780 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:58.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:04:58.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:58.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:58.782 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:58.782 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:04:58.782 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:58.782 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:58.783 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:58.783 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:04:58.784 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:58.784 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:58.784 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:04:58.784 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:58.785 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:58.785 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:58.785 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:04:58.788 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:04:58.789 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:58.790 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:04:58.790 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:04:58.791 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:04:58.792 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:04:58.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:04:58.793 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:04:58.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:04:58.793 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:04:58.794 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:03.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:03.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:03.797 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:03.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:03.839 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:04.203 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:04.204 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:04.204 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:04.204 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:04.204 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:04.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:04.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:04.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:04.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:04.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:04.210 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:04.211 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:04.212 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:04.213 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:04.213 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:04.213 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:04.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:04.213 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:04.214 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:09.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:09.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:09.216 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:09.257 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:09.258 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:09.609 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:09.609 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:09.609 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:09.609 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:09.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:09.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:09.612 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:09.613 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:09.614 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:09.615 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:09.616 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:09.617 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:09.618 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:09.619 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:09.620 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:14.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:14.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:14.622 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:14.665 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:14.666 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:15.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:15.031 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:15.032 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:15.033 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:15.034 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:15.035 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:15.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:15.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:15.035 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.035 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:15.036 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.037 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:15.038 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:15.039 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:15.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:15.039 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:15.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:15.040 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:15.041 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:20.043 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... FAIL 2020-10-09T18:05:20.112 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-10-09T18:05:20.112 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.116 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.117 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable 2020-10-09T18:05:20.117 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.117 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.117 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.118 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.118 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1114, in test_bucket_sync_disable 2020-10-09T18:05:20.118 INFO:tasks.rgw_multisite_tests: zonegroup_data_checkpoint(zonegroup_conns) 2020-10-09T18:05:20.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 367, in zonegroup_data_checkpoint 2020-10-09T18:05:20.119 INFO:tasks.rgw_multisite_tests: zone_data_checkpoint(target_conn.zone, source_conn.zone) 2020-10-09T18:05:20.119 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 359, in zone_data_checkpoint 2020-10-09T18:05:20.119 INFO:tasks.rgw_multisite_tests: (target_zone.name, source_zone.name) 2020-10-09T18:05:20.119 INFO:tasks.rgw_multisite_tests:AssertionError: failed data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-10-09T18:05:20.120 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.120 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.120 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable 2020-10-09T18:05:20.120 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.121 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.121 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.121 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.121 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1129, in test_bucket_sync_enable_right_after_disable 2020-10-09T18:05:20.122 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-09T18:05:20.122 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-09T18:05:20.122 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-09T18:05:20.122 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-09T18:05:20.123 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-09T18:05:20.123 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-09T18:05:20.123 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.123 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.124 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_sync_disable_enable 2020-10-09T18:05:20.124 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.125 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.125 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.125 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.125 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1163, in test_bucket_sync_disable_enable 2020-10-09T18:05:20.125 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-09T18:05:20.126 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-09T18:05:20.126 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-09T18:05:20.126 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-09T18:05:20.127 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-09T18:05:20.127 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-09T18:05:20.127 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.128 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.128 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_multipart_object_sync 2020-10-09T18:05:20.128 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.128 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.129 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.129 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.129 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1207, in test_multipart_object_sync 2020-10-09T18:05:20.129 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2020-10-09T18:05:20.130 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 387, in zonegroup_bucket_checkpoint 2020-10-09T18:05:20.130 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2020-10-09T18:05:20.130 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 379, in zone_bucket_checkpoint 2020-10-09T18:05:20.130 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2020-10-09T18:05:20.131 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multisite.py", line 202, in admin 2020-10-09T18:05:20.131 INFO:tasks.rgw_multisite_tests: assert r == 0 2020-10-09T18:05:20.131 INFO:tasks.rgw_multisite_tests:AssertionError 2020-10-09T18:05:20.131 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.132 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.132 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_encrypted_object_sync 2020-10-09T18:05:20.132 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.133 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.133 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.133 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.133 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1243, in test_encrypted_object_sync 2020-10-09T18:05:20.134 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-09T18:05:20.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-09T18:05:20.134 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-09T18:05:20.134 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-09T18:05:20.135 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-09T18:05:20.135 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-09T18:05:20.135 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.135 INFO:tasks.rgw_multisite_tests:====================================================================== 2020-10-09T18:05:20.136 INFO:tasks.rgw_multisite_tests:FAIL: tasks.rgw_multi.tests.test_bucket_index_log_trim 2020-10-09T18:05:20.136 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.136 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2020-10-09T18:05:20.136 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2020-10-09T18:05:20.137 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2020-10-09T18:05:20.137 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1273, in test_bucket_index_log_trim 2020-10-09T18:05:20.137 INFO:tasks.rgw_multisite_tests: cold_bucket = make_test_bucket() 2020-10-09T18:05:20.137 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 1268, in make_test_bucket 2020-10-09T18:05:20.138 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2020-10-09T18:05:20.138 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 200, in zonegroup_meta_checkpoint 2020-10-09T18:05:20.138 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2020-10-09T18:05:20.138 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-investigate-datalog/qa/tasks/rgw_multi/tests.py", line 189, in zone_meta_checkpoint 2020-10-09T18:05:20.139 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2020-10-09T18:05:20.139 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone4 2020-10-09T18:05:20.139 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.140 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-10-09T18:05:20.140 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 2815.145s 2020-10-09T18:05:20.140 INFO:tasks.rgw_multisite_tests: 2020-10-09T18:05:20.140 INFO:tasks.rgw_multisite_tests:FAILED (failures=6) 2020-10-09T18:05:20.145 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-10-09T18:05:20.146 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.154 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:20.194 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:20.194 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:20.436 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=e380f3c0-dc07-4443-880b-21f94e64f4aa 2020-10-09T18:05:20.472 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2020-10-09T18:05:20.472 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T18:05:20.473 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "0.000000" 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:20.474 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T18:05:20.475 INFO:teuthology.orchestra.run.smithi060.stdout: "last_update": "2020-10-09T17:59:53.490902Z" 2020-10-09T18:05:20.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:20.475 INFO:teuthology.orchestra.run.smithi060.stdout:] 2020-10-09T18:05:20.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-10-09T18:05:20.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.485 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:20.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:20.528 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:20.799 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:20.799 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:20.800 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:20.801 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:20.802 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:20.803 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:20.804 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:20.805 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T18:05:20.806 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:59:53.490902Z", 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:20.807 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:20.808 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:20.808 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:20.808 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:20.808 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:20.808 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:20.809 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-10-09T18:05:20.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-10-09T18:05:20.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:20.811 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:20.851 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:20.851 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:21.117 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "sync_status": { 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "info": { 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "sync", 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "num_shards": 4, 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:21.118 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "markers": [ 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 0, 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:21.119 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 1, 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:21.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 1, 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "", 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T18:05:21.121 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "0.000000", 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 2, 2020-10-09T18:05:21.122 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 2 2020-10-09T18:05:21.123 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: { 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: "key": 3, 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: "val": { 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: "state": 1, 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: "marker": "1_1602266393.490902_404.1", 2020-10-09T18:05:21.124 INFO:teuthology.orchestra.run.smithi060.stdout: "next_step_marker": "", 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: "total_entries": 0, 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: "pos": 0, 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: "timestamp": "2020-10-09T17:59:53.490902Z", 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: "realm_epoch": 4 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.125 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout: "full_sync": { 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout: "total": 1, 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout: "complete": 1 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout: } 2020-10-09T18:05:21.126 INFO:teuthology.orchestra.run.smithi060.stdout:} 2020-10-09T18:05:21.127 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-10-09T18:05:21.128 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-10-09T18:05:21.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:21.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:21.129 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:21.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:21.169 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:21.534 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:21.535 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:21.536 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:21.536 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:21.536 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:21.536 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:21.537 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:21.537 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:21.537 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:21.537 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:21.538 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:21.538 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:21.538 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:21.538 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:21.539 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:21.539 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:21.539 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.539 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:21.540 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:21.540 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:21.540 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:21.540 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:21.541 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:21.541 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:21.541 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:21.541 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:21.541 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:21.542 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:21.542 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.542 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:21.542 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:21.543 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:21.543 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:21.543 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:21.543 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:21.544 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:21.544 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:21.544 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:21.544 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:21.544 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:21.545 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.545 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:21.545 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:21.545 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:21.546 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:21.547 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:21.548 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:21.548 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:21.548 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:21.548 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:21.549 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:26.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:26.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:26.551 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:26.591 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:26.592 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:26.970 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:26.970 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:26.970 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:26.970 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:26.970 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:26.971 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:26.972 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:26.973 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.974 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:26.975 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:26.976 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:26.977 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:26.978 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:26.979 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:26.979 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:26.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:31.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:31.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:31.983 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:32.024 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:32.024 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:32.392 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:32.393 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:32.394 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:32.395 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:32.396 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:32.397 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:32.398 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:32.399 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.400 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:32.401 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:32.402 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:37.403 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:37.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:37.405 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:37.446 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:37.446 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:37.815 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:37.816 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:37.817 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:37.818 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:37.818 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:37.818 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:37.818 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:37.818 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.819 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:37.819 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:37.819 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:37.819 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:37.819 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:37.820 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:37.820 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:37.820 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:37.820 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:37.820 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:37.821 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:37.822 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:37.823 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:37.824 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:37.825 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:37.826 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:37.827 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:42.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:42.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:42.829 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:42.870 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:42.870 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:43.236 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:43.236 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:43.236 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:43.237 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:43.238 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:43.239 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:43.240 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:43.241 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:43.242 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:43.243 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:43.244 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:43.245 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:43.245 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:43.245 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:43.245 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:43.246 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:48.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:48.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:48.249 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:48.289 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:48.289 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:48.654 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:48.654 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:48.654 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:48.655 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:48.655 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:48.655 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:48.655 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:48.656 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:48.657 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:48.658 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:48.659 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:48.660 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:48.661 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:48.662 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:48.663 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:48.664 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:48.665 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:48.665 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:48.666 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:48.667 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:53.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:53.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:53.670 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:53.709 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:53.710 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:54.076 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:54.077 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:54.077 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:54.077 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:54.077 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:54.078 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:54.078 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:54.078 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:54.078 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:54.078 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:54.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:54.080 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:54.081 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:54.082 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:54.083 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:54.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:54.084 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:54.084 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:54.084 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:54.085 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:54.085 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:54.085 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:54.086 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:54.086 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:54.086 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:54.086 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:54.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.087 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.087 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:54.088 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:54.088 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:54.088 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:54.088 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:54.089 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:54.089 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:54.090 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:05:59.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:59.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:05:59.093 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:05:59.133 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:05:59.133 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:05:59.488 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:05:59.488 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:05:59.489 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:05:59.489 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:05:59.489 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:05:59.489 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:05:59.490 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:59.490 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:59.490 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:05:59.491 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:59.491 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:05:59.491 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:59.492 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:59.492 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:05:59.492 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:59.492 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:05:59.493 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:59.493 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:05:59.493 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:59.494 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.494 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:59.494 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:59.494 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:05:59.495 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:59.496 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:59.496 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:05:59.496 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:59.497 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:59.497 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:59.497 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:05:59.497 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:59.498 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:59.499 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:05:59.499 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:59.499 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:59.499 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:05:59.500 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:59.500 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:59.500 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:59.501 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:05:59.501 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:05:59.501 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.501 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:59.502 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:05:59.502 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:05:59.502 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:05:59.503 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:05:59.503 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:05:59.503 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:05:59.503 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:05:59.504 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:05:59.504 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:05:59.504 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:05:59.504 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.505 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.505 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:05:59.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:05:59.506 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:05:59.507 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:05:59.507 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:05:59.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:05:59.508 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:05:59.509 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:04.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:04.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:04.512 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:04.554 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:04.554 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:04.926 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:04.927 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:04.927 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:04.927 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:04.927 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:04.928 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:04.929 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:04.930 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:04.931 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:04.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:04.933 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:04.934 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:04.935 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:04.936 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:04.936 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:04.936 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:04.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:04.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:09.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:09.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:09.940 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:09.980 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:09.980 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:10.350 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:10.351 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:10.352 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:10.353 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:10.354 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:10.355 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:10.356 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:10.357 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.358 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:10.359 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:10.360 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:15.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:15.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:15.363 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:15.405 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:15.405 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:15.775 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:15.776 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:15.777 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:15.778 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:15.779 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:15.780 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:15.781 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:15.782 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:15.783 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:15.784 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:15.784 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:15.785 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:20.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:20.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:20.788 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:20.828 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:20.829 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:21.204 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:21.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:21.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:21.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:21.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:21.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:21.211 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.212 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:21.213 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:21.214 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:26.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:26.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:26.217 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:26.257 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:26.258 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:26.626 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:26.627 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:26.627 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:26.627 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:26.627 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:26.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:26.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:26.630 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:26.631 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:26.632 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:26.633 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:26.633 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:26.633 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:26.633 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.633 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:26.634 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:26.635 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:26.636 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:26.637 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:26.638 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:31.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:31.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:31.640 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:31.679 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:31.680 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:31.944 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:31.944 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:31.944 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:31.944 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:31.945 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:31.945 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:31.945 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:31.945 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:31.946 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:31.946 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:31.946 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:31.946 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:31.947 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:31.947 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:31.947 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:31.948 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:31.948 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:31.948 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:31.948 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:31.949 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.949 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:31.949 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:31.949 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:31.950 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:31.950 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:31.950 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:31.951 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:31.951 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:31.951 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:31.951 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:31.952 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:31.952 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.952 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:31.953 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:31.953 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:31.953 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:31.954 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:31.954 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:31.954 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:31.955 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:31.955 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:31.955 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:31.956 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:31.956 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.956 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:31.957 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:31.958 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:31.959 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:31.959 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:31.959 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:31.959 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:31.960 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:36.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:36.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:36.963 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:37.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:37.004 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:37.368 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:37.369 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:37.370 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:37.371 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:37.372 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:37.373 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:37.374 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:37.375 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.376 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:37.377 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:37.378 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:42.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:42.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:42.381 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:42.422 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:42.422 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:42.789 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:42.789 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:42.789 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:42.790 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:42.790 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:42.790 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:42.790 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:42.790 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:42.791 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:42.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:42.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:42.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:42.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:42.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:42.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:42.798 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:42.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:42.798 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:42.798 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:42.798 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:42.799 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:47.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:47.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:47.802 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:47.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:47.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:48.202 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:48.202 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:48.203 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:48.204 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:48.205 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:48.206 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:48.207 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.208 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:48.209 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:48.210 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:48.211 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:48.211 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:48.211 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:48.211 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:48.211 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:48.212 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:53.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:53.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:53.214 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:53.252 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:53.253 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:53.620 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:53.621 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:53.622 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:53.623 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:53.624 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:53.625 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:53.626 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:53.627 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:53.628 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:53.629 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:53.630 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:53.630 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:53.630 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:53.631 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:06:58.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:58.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:06:58.634 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:06:58.674 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:06:58.675 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:06:59.044 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:06:59.045 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:06:59.045 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:06:59.045 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:06:59.045 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:06:59.045 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:59.046 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:59.047 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:59.048 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:59.049 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:59.050 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:06:59.051 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.052 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:06:59.053 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:06:59.054 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:04.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:04.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:04.058 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:04.111 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:04.112 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:04.469 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:04.469 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:04.469 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:04.469 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:04.469 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:04.470 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:04.471 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:04.472 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.473 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:04.474 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:04.475 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:04.476 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:04.477 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:04.478 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:04.478 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:04.478 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:04.479 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:09.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:09.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:09.482 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:09.520 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:09.521 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:09.893 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:09.894 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:09.895 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:09.896 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:09.897 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:09.898 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:09.899 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:09.900 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:09.901 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:09.902 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:09.903 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:09.903 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:09.903 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:09.903 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:09.903 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:09.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:14.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:14.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:14.906 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:14.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:14.949 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:15.304 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:15.304 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:15.305 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:15.306 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.307 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:15.308 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:15.309 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:15.310 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:15.311 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:15.312 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:15.313 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:15.314 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:15.314 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:15.315 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:20.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:20.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:20.317 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:20.357 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:20.357 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:20.724 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:20.725 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:20.725 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:20.725 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:20.725 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:20.725 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:20.726 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:20.727 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.728 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:20.728 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:20.728 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:20.728 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:20.729 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:20.729 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:20.729 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:20.730 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:20.730 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:20.730 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:20.730 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:20.731 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.731 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:20.731 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:20.732 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:20.733 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:20.734 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:20.734 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:20.734 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.734 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:20.735 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:20.735 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:20.735 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:20.735 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:20.736 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:20.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.737 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.737 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:20.737 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:20.738 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:20.738 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:20.738 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:20.739 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:20.739 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:20.740 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:25.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:25.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:25.742 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:25.781 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:25.781 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:26.150 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:26.150 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:26.151 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:26.152 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:26.153 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:26.154 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:26.155 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:26.156 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:26.157 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:26.158 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:26.161 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:26.162 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:26.162 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:26.163 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:31.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:31.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:31.166 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:31.208 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:31.209 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:31.574 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:31.574 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:31.575 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:31.576 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:31.577 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:31.578 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:31.579 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:31.580 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:31.581 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:31.584 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.585 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:31.586 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:31.587 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:31.588 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:31.588 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:31.590 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:36.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:36.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:36.593 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:36.634 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:36.635 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:37.002 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:37.003 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:37.003 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:37.004 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:37.004 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:37.004 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:37.004 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:37.005 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:37.005 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:37.005 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:37.006 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:37.006 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:37.006 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:37.006 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:37.007 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:37.007 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:37.007 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:37.007 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:37.008 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:37.008 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.008 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:37.009 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:37.009 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:37.009 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:37.009 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:37.010 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:37.010 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:37.010 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:37.011 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:37.011 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:37.011 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:37.011 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.012 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:37.012 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:37.012 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:37.013 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:37.013 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:37.013 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:37.013 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:37.014 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:37.014 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:37.014 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:37.015 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:37.015 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.016 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:37.016 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:37.016 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:37.016 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:37.016 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:37.017 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:37.017 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:37.017 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:37.017 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:37.017 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:37.018 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:37.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.018 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.018 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:37.018 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:37.019 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:37.019 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:37.020 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:42.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:42.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:42.023 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:42.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:42.062 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:42.432 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:42.432 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:42.432 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:42.433 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:42.434 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:42.435 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:42.436 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:42.437 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:42.438 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.439 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:42.440 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:42.441 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:42.442 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:42.442 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:42.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:47.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:47.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:47.446 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:47.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:47.489 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:47.860 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:47.860 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:47.860 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:47.860 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:47.861 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:47.862 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:47.863 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:47.864 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:47.865 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:47.866 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:47.866 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:47.866 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:47.866 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:47.867 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:47.868 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:47.869 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:47.870 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:47.871 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:52.872 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:52.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:52.873 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:52.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:52.914 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:53.283 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:53.284 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:53.284 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:53.284 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:53.284 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:53.285 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:53.286 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:53.287 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:53.288 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:53.288 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:53.288 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:53.288 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:53.288 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:53.289 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:53.290 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:53.290 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:53.290 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:53.290 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:53.291 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:53.291 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:53.291 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.291 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:53.292 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:53.292 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:53.292 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:53.292 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:53.293 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:53.293 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:53.293 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:53.293 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:53.294 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:53.294 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:53.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.294 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.295 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:53.295 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:53.295 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:53.295 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:53.296 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:53.296 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:53.296 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:53.298 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:07:58.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:58.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:07:58.301 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:07:58.340 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:07:58.341 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:07:58.708 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:07:58.708 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:58.709 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:07:58.710 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:58.711 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:58.712 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:07:58.713 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:07:58.714 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:07:58.715 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.716 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:07:58.717 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:07:58.718 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:03.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:03.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:03.721 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:03.760 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:03.761 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:04.129 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:04.129 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:04.129 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:04.129 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:04.130 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:04.131 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:04.131 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:04.131 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:04.131 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:04.131 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.132 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:04.133 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:04.134 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:04.134 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:04.134 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:04.135 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:04.135 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.135 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:04.135 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:04.136 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:04.136 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:04.136 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:04.137 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:04.137 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:04.137 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:04.137 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:04.138 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:04.139 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:04.140 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:04.141 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:04.141 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:04.142 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:09.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:09.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:09.144 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:09.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:09.177 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:09.639 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:09.640 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:09.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:09.643 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:09.644 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:09.645 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:09.646 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.647 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:09.648 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:09.649 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:14.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:14.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:14.652 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:14.696 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:14.696 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:15.067 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:15.067 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:15.067 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:15.068 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:15.068 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:15.068 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:15.069 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:15.069 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:15.069 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:15.070 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:15.070 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:15.070 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:15.070 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:15.071 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:15.071 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:15.071 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:15.072 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:15.072 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:15.072 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:15.073 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.073 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:15.073 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:15.073 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:15.074 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:15.074 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:15.074 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:15.075 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:15.075 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:15.075 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:15.076 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:15.077 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:15.078 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:15.079 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:15.080 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.080 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:15.080 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:15.081 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:15.081 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:15.081 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:15.081 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:15.082 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:15.082 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:15.082 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:15.083 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:15.083 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:15.083 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.084 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.084 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:15.084 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:15.084 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:15.085 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:15.085 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:15.085 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:15.085 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:15.087 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:20.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:20.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:20.090 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:20.129 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:20.130 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:20.503 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:20.503 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:20.504 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:20.504 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:20.504 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:20.505 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:20.506 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:20.506 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:20.506 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:20.506 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:20.506 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:20.507 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:20.508 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:20.509 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:20.510 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:20.511 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:20.512 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:20.513 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:20.514 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:20.514 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:20.514 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:20.514 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:20.515 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:25.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:25.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:25.518 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:25.559 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:25.559 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:25.922 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:25.922 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:25.923 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:25.924 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.925 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:25.926 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.927 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:25.928 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:25.929 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:25.930 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:25.931 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:25.932 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:25.933 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.934 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:25.935 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:25.936 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:25.937 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:30.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:30.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:30.939 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:30.972 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:30.973 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:31.340 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:31.340 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:31.341 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:31.342 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:31.343 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:31.344 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:31.345 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:31.346 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:31.347 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:31.348 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:31.349 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:31.350 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:31.351 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:36.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:36.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:36.353 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:36.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:36.393 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:36.765 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:36.766 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:36.767 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.768 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:36.769 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:36.770 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:36.771 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:36.772 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:36.773 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:36.774 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:36.775 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:36.775 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:36.776 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:41.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:41.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:41.779 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:41.821 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:41.821 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:42.179 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:42.180 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:42.181 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.182 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:42.183 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:42.184 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:42.185 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:42.186 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:42.188 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:42.189 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:42.190 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:42.190 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:42.190 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:42.190 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:42.191 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:47.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:47.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:47.194 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:47.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:47.234 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:47.603 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:47.603 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:47.604 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:47.605 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:47.606 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:47.607 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:47.608 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:47.609 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:47.610 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.611 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:47.612 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:47.613 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:52.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:52.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:52.616 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:52.657 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:52.658 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:53.030 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:53.031 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:53.032 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:53.032 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:53.032 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:53.032 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:53.032 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:53.033 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:53.034 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:53.035 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:53.036 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.036 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:53.036 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:53.036 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:53.036 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:53.037 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:53.038 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:53.039 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:53.040 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:53.041 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:53.042 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:08:58.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:58.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:08:58.045 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:08:58.085 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:08:58.086 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:08:58.445 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:08:58.447 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:08:58.448 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:08:58.449 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:08:58.450 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:08:58.452 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:08:58.453 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:58.454 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:58.455 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:08:58.456 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:58.457 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:08:58.458 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:58.458 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:58.459 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:08:58.459 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:58.459 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:08:58.459 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:58.460 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:08:58.460 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:58.460 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.460 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:58.461 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:58.461 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:08:58.461 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:58.461 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:58.462 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:08:58.462 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:58.462 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:58.463 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:58.463 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:08:58.463 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:58.464 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.464 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:58.464 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:58.464 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:08:58.465 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:58.465 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:58.465 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:08:58.465 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:58.466 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:58.466 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:58.466 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:08:58.467 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:08:58.467 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.467 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:58.467 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:08:58.468 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:08:58.468 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:08:58.468 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:08:58.469 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:08:58.469 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:08:58.469 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:08:58.469 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:08:58.470 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:08:58.470 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:08:58.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.470 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.471 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:08:58.471 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:08:58.471 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:08:58.472 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:08:58.472 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:08:58.472 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:08:58.472 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:08:58.475 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:03.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:03.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:03.478 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:03.517 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:03.518 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:03.790 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:03.791 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:03.792 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.793 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:03.794 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:03.795 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:03.796 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:03.797 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:03.799 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:03.800 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:03.800 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:03.801 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:08.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:08.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:08.804 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:08.844 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:08.845 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:09.210 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:09.210 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:09.210 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:09.211 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:09.212 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:09.213 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:09.214 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:09.214 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:09.214 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:09.214 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:09.214 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:09.215 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:09.216 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:09.217 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:09.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:09.219 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:09.220 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:09.220 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:09.220 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:09.221 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:14.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:14.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:14.223 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:14.266 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:14.267 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:14.636 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:14.636 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:14.637 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:14.638 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:14.639 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:14.640 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:14.641 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.642 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:14.643 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.644 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:14.645 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:14.646 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:19.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:19.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:19.648 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:19.688 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:19.689 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:20.053 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:20.054 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:20.054 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:20.054 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:20.054 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:20.055 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:20.056 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:20.057 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:20.058 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:20.059 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.060 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:20.061 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:20.062 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:20.063 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:20.063 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:20.063 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:20.063 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:20.064 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:25.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:25.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:25.067 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:25.108 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:25.109 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:25.380 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:25.380 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:25.380 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:25.380 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:25.381 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:25.382 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:25.383 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:25.384 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:25.385 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:25.386 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:25.387 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.388 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:25.389 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:25.390 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:30.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:30.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:30.393 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:30.434 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:30.434 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:30.797 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:30.797 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:30.798 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:30.799 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:30.800 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:30.801 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:30.802 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:30.803 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:30.804 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:30.805 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:30.806 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:30.807 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:30.807 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:30.808 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:35.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:35.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:35.811 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:35.852 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:35.852 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:36.215 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:36.215 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:36.215 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:36.215 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:36.216 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:36.217 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:36.217 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:36.217 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:36.217 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:36.218 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:36.218 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:36.218 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:36.219 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:36.219 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.219 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:36.219 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:36.220 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:36.220 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:36.220 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:36.220 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:36.221 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:36.221 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:36.221 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.222 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:36.223 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:36.224 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:36.225 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:36.226 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:36.227 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:36.228 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:36.229 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:36.229 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:36.230 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:41.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:41.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:41.234 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:41.297 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:41.297 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:41.659 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:41.660 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:41.661 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:41.662 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:41.663 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:41.664 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:41.665 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:41.667 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:41.668 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:41.669 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:41.670 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:41.671 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:41.672 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:41.673 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.674 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:41.676 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.677 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:41.678 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:41.679 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:46.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:46.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:46.682 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:46.725 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:46.725 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:46.993 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:46.993 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:46.993 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:46.993 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:46.993 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:46.994 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:46.995 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:46.996 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:46.997 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:46.998 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:46.999 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:47.000 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:47.001 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:47.002 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:47.003 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:52.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:52.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:52.006 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:52.045 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:52.046 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-09T18:09:52.409 INFO:teuthology.orchestra.run.smithi184.stdout:{ 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "sync_status": { 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "info": { 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "status": "sync", 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "num_shards": 4, 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "period": "e380f3c0-dc07-4443-880b-21f94e64f4aa", 2020-10-09T18:09:52.410 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: "markers": [ 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 0, 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:52.411 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263785.872885_64.1", 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 1, 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:16:25.872885Z", 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.412 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 1, 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "", 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:52.413 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "0.000000", 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 2, 2020-10-09T18:09:52.414 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602263917.851306_97.1", 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:18:37.851306Z", 2020-10-09T18:09:52.415 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 2 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: { 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: "key": 3, 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: "val": { 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: "state": 1, 2020-10-09T18:09:52.416 INFO:teuthology.orchestra.run.smithi184.stdout: "marker": "1_1602265031.065489_324.1", 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: "next_step_marker": "", 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: "total_entries": 0, 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: "pos": 0, 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: "timestamp": "2020-10-09T17:37:11.065489Z", 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: "realm_epoch": 4 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.417 INFO:teuthology.orchestra.run.smithi184.stdout: ] 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout: }, 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout: "full_sync": { 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout: "total": 1, 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout: "complete": 1 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout: } 2020-10-09T18:09:52.418 INFO:teuthology.orchestra.run.smithi184.stdout:} 2020-10-09T18:09:52.419 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1602266393.490902_404.1 target=1_1602265031.065489_324.1 2020-10-09T18:09:57.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:57.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-10-09T18:09:57.422 INFO:teuthology.orchestra.run.smithi184:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-10-09T18:09:57.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setuser ceph since I am not root 2020-10-09T18:09:57.461 INFO:teuthology.orchestra.run.smithi184.stderr:ignoring --setgroup ceph since I am not root 2020-10-10T05:01:37.445 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-10-10T05:01:37.478 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-10-10T05:01:37.479 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2020-10-10T05:01:37.479 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-10-10T05:01:37.480 DEBUG:teuthology.task.console_log:Killing console logger for smithi184 2020-10-10T05:01:37.480 DEBUG:teuthology.exit:Finished running handlers